WO2006091928A2 - Digital video identification and content estimation system and method - Google Patents

Abstract

A method and system are provided for determining a source optical system of still image data or video image data. An optical system is operated to identify system specific artifacts in image data produced by the system. The system specific artifacts are characterized by pattern recognition software. A comparison of the pattern of the known system, and preferably patterns of a plurality of known systems, are compared to image data of unknown sources to permit matching of the patterns and thereby identification of the source optical system. Alternatively, the comparison can determine if two sets of image data have a common source, or if a portion of image data has been manipulated.

Description

S P E C I F I C A T I O N TITLE

"DIGITAL VIDEO IDENTIFICATION AND CONTENT ESTIMATION SYSTEM

AND METHOD"

CROSS-REFERENCE TO RELATED APPLICATION

[0001] This application claims the benefit of United States Provisional Patent

Application Serial No. 60/655,406, filed February 24, 2005, which is incorporated herein by reference.

BACKGROUND OF THE INVENTION Field of the Invention

[0002] The present invention relates generally to a system and method for identifying a source of still image data and video image data.

Description of the Related Art

[0003] Often in the investigative search and recovery efforts of law enforcement and the military, recorded visual communications data is discovered in the form of tapes (e.g., VHS, miniDV, or BetaCam) or discs (e.g., hard drives, CDs, or DVDs). Visual communication data (VCD) can be captured by number of different devices such as scanners, analog or digital camcorders, film or digital cameras. Beyond the visual content of the recovered evidence, the source of the VCD content is a vital forensic clue potentially revealing the origin, behavior, and supply chain of a suspect individual or organization. On occasion when VCD sources (cameras or scanners) and VCD media (tapes or discs) may be found on-site, one would like to verify that the recovered source device is the "smoking camera" that recorded the data found on the recovered media. The type, make, or model of the original image capture device, storage or transmission methods and evidence of VCD manipulation data are all examples of critical VCD information that presently is difficult if not impossible to obtain with current investigation methods in absence of on-site physical evidence (i.e., fingerprints, receipts, or financial records).

[0004] It is highly desirable for law enforcement and the military to be able to automatically identify or verify otherwise anonymous recordings by processing their visual content. Being able to determine the source (or history) of an image or video sequence is critical; source identification deepens an investigation by:

[0005] • Unveiling a potentially traceable electronic path of the visual communication data.

[0006] * Assisting in the identification of the supply chain of a suspect.

[0007] • Establish time frames of VCD.

[0008] • Providing evidence of corruption or modification of VCD content.

[0009] • Predicting potential failure or malfunction of a critical digital imaging system.

[0010] Current methods to authenticate and verify image and video content involve the use of watermarking techniques. Watermarking is the purposeful introduction of a (message) signal to an image (sequence) for the function of uniquely identifying the owner or source of the digital work. This watermark signal is subtly embedded into the image (sequence) to later prove ownership or origin of the content. The problem with such watermarking methods in the context of forensics is that the mark must be intentionally or actively introduced to VCD content. The embedding of watermarks by a suspect is unlikely to be done intentionally unless a message is stored in the mark. A sophisticated individual or organization would avoid equipment that would purposefully introduce a watermark.

[0011] The field of forensic investigation is in need of a passive, content independent

VCD source identification and verification solution. Passive detection enables device verification and identification without an intentional watermark. An example of the successful use of passive watermarks in forensics is found in ballistics. In a specific ballistics test, a recovered gun is loaded and fired into a target. The markings on the cartridge and projectile are recorded. The images of these markings are matched to casings and projectiles found at the crime scene. If the markings on each set of projectiles match, the weapon is associated with the crime scene. This method has been successfully used as evidence in high profile murder cases.

[0012] Source device distortions occur in obtaining image data. In the process of capturing a spatio-temporal representation of a scene, practical imaging systems invariably degrade the signal representing the original scene. Device physics, optics, camera design, environment, electro-magnetic radiation and post-processing methods all contribute to the ultimate quality and content of the final image (sequence). Image (sequence) degradations can be traced to three causes, 1) the capture device, 2) downstream signal processing procedures, and 3) lighting and environment.

[0013] An example of the types of distortions introduced by an image capture system can be seen in the case of a conventional digital video camera based on a charge coupled device (CCD). In the digital camcorder, light is focused on a focal plane array of photo-gates developing a photo-electronic charge proportional to the intensity of the incident light in the potential well at each pixel. The photo-electronic current is not the only contribution to the final pixel response. Figure 1 shows that a number of distortions are introduced to the original signal by the digital video camera. In particular, light 10 enters a lens system 12 and is detected by a detector 14. The detector 14 is, in one embodiment, a digital detector, such as a CCD detector, is used to obtain the digital image. Dark current 16, photon and dark current shot noise 18 and 20, electro-magnetic interference 22, and photo response non- uniformity 24 all contribute and corrupt the incident photo-electronic charge signal 25. Charge to voltage conversion 26 and readout strategies used to obtain a two dimensional representation of the image introduce additional distortions such reset noise 28, amplifier noise 30, and analog-to-digital (quantization) noise 32. The aforementioned distortions are physically and empirically related to pixel array design, electronic circuitry and the semiconductor manufacturing process. A component of these distortions are also known to vary slightly with temperature.

[0014] After digitization, a host of signal post-processing methods are used to reduce the effect of these distortions and to prepare the image for viewing by the end user. Postprocessing methods such as auto white balance, noise reduction, non-uniformity correction 34, color filter inversion and lossy compression 36, all introduce additional signal processing noises to the original signal.

[0015] From this example, it is clear that the original signal 25 is corrupted (or shaped) by the image acquisition process. This corruption is spatial and temporal in nature. The level of shaping or corruption is determined by camera design, and device physics and signal post-processing methods.

[0016] Residual source distortions are being investigated by the present inventor as a passive watermark symbol. It is not possible for current post-processing methods to completely remove the distortions caused by electronic or signal processing noise. The objective of most image post-processing methods is to reduce distortions to a point below a visual threshold under normal viewing conditions. The result is a residual (mathematical) distortion that represents a potentially strong watermark signal that is passively embedded in VCD.

[0017] This passive watermark signal has several important properties that make it a solid candidate as a watermark for the identification and verification of VCD sources, for example residual distortions:

[0018] • Have components are invisible or partially visible to the human eye during normal viewing.

[0019] • Are descriptive signatures regarding the scene and the VCD capture device. [0020] • Are composed of device dependent, system dependent and signal dependent components.

[0021] The uniqueness and device/system dependent of this is signal is a critical characteristic needed in the identification and verification of a VCD source. We are concerned with two classes of uniquely identifiable signature information in this work, 1) device specific information and 2) model specific information.

[0022] Evidence of passively embedded device specific information has been found.

In particular, evidence of unique device specific information embedded in VCD has been found recently in the field of forensics. Japanese and Dutch researchers, in a test of nine separate cameras, found that averaging one-hundred (100) images of a uniform dark scene reveals a distinct pattern corresponding to the non-homogenous nature of the dark current in a CCD array. This pattern, referred to as fixed pattern noise above, is the result of randomness and defects in the CCD manufacturing process. These patterns are device specific, temperature sensitive spatial distortions that vary from array to array within a camera model. Currently available methods require a sequence of (uniform) dark images to detect these patterns and have been only applied to CCD cameras according to the publicly available literature at this time.

[0023] Evidence of passively embedded model specific information has also been found. In particular, evidence of unique, model specific information embedded in VCD is indicated in the field of image enhancement and restoration. The goal of image enhancement or restoration is to isolate and remove distortions that have been introduced to recorded (video) images. At no other time has the demand to correct or enhance images been greater than in the current age of digital photography. In general, image correction and enhancement is a tedious, trial and error task. Recently, mass market digital imaging software makers have sought to develop "one-click" image correction solutions to assist users in optimizing the quality of their pictures. The industry has found that a "one-size fits all" solution to image correction is limited in its effectiveness. Additional knowledge regarding the camera (and the scene) is needed to accurately correct an image. Some software makers such as Adobe Systems and nik Multimedia (now NIK Software) are beginning to develop and utilize camera specific profiles that describe special (proprietary) characteristics of the camera in order to assist enhancement software in the reduction of noise and color distortions. The need for additional profiles, beyond the meta-data provided by standard image exchange formats such as EXIF (www.exif.org), is significant evidence of the variability in model dependent characteristics across cameras in the marketplace. This variability is also present in video cameras.

[0024] Camera differentiation has been found. According to GlobalSpec, more than fifty camera component suppliers (CCSs) supply components to an estimated one-hundred and fifty digital camera original equipment manufacturers (OEMs) to produce more than fifty million (consumer) still and video cameras each year, worldwide. Some OEMs such as Kodak and Sony also serve as their own CCSs with in-house chip design/fabrication capabilities. Each OEM, has a specific level of expertise in camera design and a targeted market focus. In consumer still and video cameras, the top five are responsible for the more than eighty percent of the unit volume.

[0025] Camera OEMs differentiate themselves through the strategic development or selection of a camera reference design (CRD). A CRD minimally consists of a sensor, optical configuration, and post-processing components. When a sensor technology and optical configuration is determined, the baseline size, resolution, color fidelity, noise and sensitivity characteristics for the device are established. Post-processing methods are implemented to enhance the baseline and provide the final image to the end user. The class and complexity of post processing methods such as color interpolation, noise reduction, gamma correction and contrast enhancement vary widely among camera makers and is one of the highly competitive differentiating factors in the camera industry. Despite the extensive marketing of megapixels, it is known in the industry for some time that post-processing ultimately determines the quality of the captured image, not just the number of pixels in the camera. The intense competition in the industry and the adoption of emerging industry standards makes post- processing, sensor and optical design a model level differentiator in digital still and video cameras.

SUMMARY OF THE INVENTION

[0026] The present invention provides a system and method for identifying a source of still image data or video image data by determining effects that optical, environmental, electrical and/or mechanical aspects of the device have on the resulting still or video image data. Existing still or video image data is examined for the presence of the effects of these optical, environmental, electrical and/or mechanical aspects so as that the source imaging system is identified.

[0027] Aspects of the present invention provide a passive content-independent video source identification and verification solution to assist the forensic investigator in the analysis of recovered visual communications data. Passive source identification in imaging requires that unique device dependent "markings" are present in VCD (visual communications data). This is provided according to the preferred embodiments of the present invention, wherein source detection can be done independently of image content.

BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 is a schematic illustration of a processing path of the original photo- electronic current signal i_ph, at a pixel site in a digital video camera which is corrupted by several distortions directly related to device physics and signal post-processing methods;

[0029] Figure 2 is a front perspective view of a system for performing video source analysis for source identification and verification of visual communications data according to the principles of the present invention, and showing the components of the system;

[0030] Figure 3 is a block diagram of source analysis software that performs source identification and verification by pattern estimation and classification using pattern extraction/segmentation, feature extraction and feature classification; [0031] Figures 4a, 4b, 4c and 4d are examples of images and spatio-temporal patterns (contrast enhanced spatial patterns shown here) that can be extracted for further analysis according to the principles of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0032] The present system and method provides a passive watermark detection for image data that may be used to identify a source of the image data, identify whether image data was generated by a common source, or identify if manipulation of the image data has occurred in an image set or image stream. The method and system uses noise and artifacts inherent in the image generating device as the passive watermark in the image data. The image data may be still image data or video image data. The source of the image data may be any imaging system, including digital cameras, video cameras, webcams, satellite and orbital imaging systems, terrestrial imaging systems, film cameras and film video cameras, analog and digital video cameras, image scanners, copiers, facsimile machines, security cameras, CCD devices, chip based imaging devices, linear array based imaging systems, focal plane array based imaging systems, and the like.

[0033] The imaging devices produce an image of a scene or object and the image produced includes elements that are specific to the scene or object and elements that are specific to the imaging system. The present method and system separates those elements that are specific to the imaging system from the elements specific to the scene or object and uses the imaging system specific elements as a source identifier. The source imaging system may be available for study so that the source specific elements are generated by the imaging system for comparison, but that is not necessary in every case. It is also possible to derive the source specific elements from image data obtained from known or even unknown sources as a basis for the comparison.

[0034] According to preferred embodiments, a video source analysis system and method is provided that includes an engineering workstation 50 running source analysis software, a bank of calibrated media playback sources 52, 54, 56 and 58 and a "dark box" enclosure 60 as shown in Figure 2. The engineering workstation 50 of one embodiment is an Intel based PC running the Windows XP operating system. In the illustration, the workstation 50 includes a keyboard, display and CPU housing. A mouse or other pointing device is also preferably provided. The workstation may instead be a portable computer, such as a laptop computer, or some other type of computer. The calibrated media playback sources, which here are a VHS tape player 52, a CD/DVD player 54, a mini-DV player 56, and a Betacam player 58, provide image sequence data for processing by the source analysis software through a PC interface (e.g., IEEE-1394, Camera-Link, or SDI). These sources 52 - 58 are calibrated in software to subtract out and adjust for the response of the playback devices in the analysis. The calibrated sources for CD/DVD, VHS, MiniDV, and BetaCam media shown in Figure 2 are just one example of players for the image media, and other types of media players may also be provided, as needed. The "dark box" 60 provides a light shielded enclosure for examining a fixed pattern noise and defects specific to an image capture device for the source verification mode. An optical system, such as a video camera, still camera, webcam, CCD imaging system, security camera, or other image capture device are placed in the dark box and operated to generate images of the artifacts inherent in the camera system, such as dark current, readout current and other noise. This information is collected and used by the present software to identify fixed pattern noise that is characteristic of the optical system. Further details of the "dark box" enclosure are provided below.

[0035] It is also possible that other techniques may be used to generate the fixed pattern noise of the optical system, such as placing a lens cover over the lens system, directing the camera system to a uniform surface and uniformly lit surface, or other technique to detect characteristics, distortions, noise and artifacts of the lens system, image capture device, image read-out circuitry and post-processing circuitry of the imaging system.

[0036] The present method and system can use any characteristic of the image data that specifies or at least indicates a source or possible source of the image data, including optical distortions of the lens and other optical components; variations in pixel sensitivity (sometimes referred to as hot pixels and cold pixels); dark current; system noise; electrical characteristics of the image read out, data handling and data storage in the camera system; post image acquisition processing, image transmission artifacts, and the like. Steady state and regularly occurring transient aspects may be considered. Further effects and artifacts in the image data that are used in the present system and method include temperature and other environmental effects, as well as effects of signal processing subsystems in the camera. The effects of the environment and downstream signal processing is used to distinguish one device from another.

[0037] According to preferred aspects of the present method, the source analysis software segments, detects, learns, and classifies VCD (video communications data) data patterns to identify and verify VCD source devices as shown in Figure 3. The present source analysis software processes the image sequence in a series of frame intervals. In particular, an observable scene 70 is sensed as image data by a sensor or camera system, the output of which is either input directly to the software or the output of the camera is recorded onto a recordable media and the image data retrieved from the recorded media, either of which is the image data source 72. The image data from the data source 72 is provided to the software. Algorithms in the software treat source identification and verification as a pattern estimation and classification problem. Image sequence data from the remote sensors or recorded media is processed for pattern extraction/segmentation 76, feature extraction 78 and feature classification 80. The software allows the user to specify additional prior knowledge to the software at each stage, as indicated at 82. Preprocessing 74 and post processing 84 may also be provided.

[0038] Patterns from passive watermark signatures are part of the source identification and verification feature space. The pattern classification used in the present method and system is based on technology that is becoming more mature and reliable. The present embodiments leverage recent understandings in visual pattern classification and signal processing to differentiate seemingly similar patterns based on statistical methods. For example, studies by Julesz in visual pattern classification observed that the human visual system (HVS) relies primarily on the first two moments of random dot patterns for pattern discrimination. Textures whose first two moments are sufficiently similar but yet have different higher order moments are in distinguishable by the HVS. This is not the case with the preferred embodiments, wherein is used higher order statistical methods to analyze and differentiate patterns. A key aspect of some embodiments is an ability to interpret higher order statistics of a signal processed by a complex system.

[0039] The present method utilizes a class of techniques to extract camera and model specific passive watermark signals from the visual content. The method models the spatial and temporal variation of the image at each pixel as a piecewise constant function plus a random variation. The constant over an interval is adapted to account for changes related pixel illumination due to lighting and/or motion. This is achieved using adaptive filtering with change detection (Gustafson 2000, Basseville and Nikiforov 1993, Haykin 2001). This method has the advantage of allowing the accurate measurement of pixel transients without introducing artifacts from motion estimation and compensation errors. Furthermore, this method allows the spatial and temporal variability of each pixel to be explicitly measured and modeled. The goal of this method is to separate the image content from the signal(s) introduced by the system and environment. The residual random variation contains a spatial and temporal pattern representing the effects of the sensor, downstream signal processing and environment. An example of a signal extraction method is shown for two sample images in Figures 4a and 4c. The spatial pattern shown here contains (separable) signal dependent and signal independent components that are visually transparent to the human viewer under normal viewing conditions. The signal independent patterns are shown in Figures 4b and 4d. Experimentation with this method shows that such patterns are found in every digitized image. According to the present method, a pattern analysis and a sufficient feature space for source identification and verification are provided. The developed spatio- temporal patterns are subject to a parameterized model based on the cumulants and moments of the random variation of the pattern (Duda et. al 2001, Pratt 2001). These model parameters are also stored with the pattern for comparison. [0040] The present system and method provide a passive image content authentication and verification solution that provides critical forensic information to the investigator/analyst desktop. Features of embodiments of the present method include:

[0041] • Content independent passive watermark detection,

[0042] • Device identification - type, make, model, and year,

[0043] • Smoking Camera™ source device verification mode (an adaptation of the present method in which a specific camera system is identified as a source of image data.

[0044] • Splice/Fault detection - modified image content detection mode

[0045] • Calibrated playback sources for analog media.

[0046] • Incorporates user prior knowledge in authentication/verification.

[0047] • Expandable source device feature space.

[0048] • International analog video standards support

NTSC/PAL/SECAM/HpTV).

[0049] The present system includes various operating modes and in the preferred embodiment of the video source analysis system three main modes, namely a verification mode, an identification mode, and a fault detection mode.

[0050] In verification mode, the present system and method addresses two important questions, 1) "Did this camera take this video?" and 2) "Are these (n) videos from the same camera?". To answer the first question, a suspect camera 62 is placed in the "dark box" enclosure 60 shown in Figure 2. An extended image sequence is taken from this camera 62 under these conditions to determine the camera fingerprint. Prior work in this area determined camera fingerprints based on the spatial non-uniformity of dark current in the array. The present method implements an extension of the existing CCD fingerprinting methods to capture a spatio-temporal non-uniformity pattern of the array 14 (see Figure 1). Once the pattern (message) is estimated from the camera, the verification problem becomes a watermark detection problem where a number of algorithms are applied to detect the pattern in the input sequence. The verification result will be a (true or false) decision and a confidence estimate.

[0051] To answer question 2), passive watermark signatures are estimated from each sequence and compared using pattern classification measures. This capability is an extension of existing array fingerprint estimation techniques to the case of non-dark scenes. This approach is based on passive watermark signature estimation techniques like those demonstrated in Figures 4a, 4b, 4c and 4d. The result will be a (true/false) decision and a confidence estimate.

[0052] In identification mode, the present software algorithms process multiple frame intervals to extract the passive watermark signal of the image sequence, calculate the corresponding feature vector and estimate the most likely source from previously learned cameras in the system database. A feature vector here is a series of vectors connecting one distinctive feature in the image to another, much the same as in fingerprint identification. Using the analysis software in this mode allows the investigator to interactively monitor the evolution, or progress, of the source estimate and incorporate additional prior knowledge regarding the VCD in the identification process. The analysis software display results of the classification including the most likely camera profiles and confidence intervals. The present system is trained with an existing camera profile database to provide the identification. The camera profile database can be expanded and updated to accommodate new cameras that enter the market, new cameras that are added to a system or that are being considered in an investigation, or to accommodate a user's requirements.

[0053] In a fault detection mode, the present system searches for inconsistencies or changes in the passive watermark signal that indicate tampering or modification of the video image sequence. Significant changes in the passive watermark over the course of the image sequence may indicate:

[0054] • Splicing or, the incorporation of data from another source;

[0055] • Additional post-processing/manipulation;

[0056] • Failure of the sensor.

[0057] Fault detection mode processes an interval of frames from the input image sequence to estimate a passive watermark signature. The estimated signature is compared to signatures estimated from (overlapping) intervals of the same length to detect changes in the signal. When a change is detected, the frame interval is tagged as an outlier for further analysis by the investigator/analyst.

[0058] The digital video identification and content estimation software incorporates methods for analyzing visual communications data (VCD)₅ images and video, such as those obtained from law enforcement or military operations. Specifically, the present system leverages recent advances in signal processing and understandings of the human visual system to provide an automatic content independent source identification and verification of VCD. Using the present system and method increases the speed and decreases the cost of investigative operations.

[0059] Thus, there is shown and described a system and method for identifying source specific artifacts in image data and comparing the source specific artifacts of a first image data set to source specific artifacts of a second image data set. The comparison enables identification of a specific source camera system for an image data set. The comparison also enables identification of splicing, processing or other manipulation of portions on an image sequence or video. The comparison also enables determination of whether a first image set has a same source as a second image set, regardless of whether the sources are known. The present method and system are particularly useful in forensic investigations or anytime that information is desired about particular image data. [0060] Since the present method and system provides a passive watermark system, the passive watermark technique may be used to identify source data for copyright matters, settle plagiarism issues, provide authorship verification, and the like.

[0061] Although other modifications and changes may be suggested by those skilled in the art, it is the intention of the inventors to embody within the patent warranted hereon all changes and modifications as reasonably and properly come within the scope of their contribution to the art.

Claims

I CLAIM:

1. A method for identifying a source of image data, comprising the steps of: operating a camera system to generate known image data; analyzing the known image data for at least one of patterns and models of patterns that are indicative of the camera system; and comparing the patterns to unknown image data so as to determine if the unknown image data was generated by the camera system.

2. A method as claimed in claim 1, further comprising the steps of: storing a plurality of at least one of patterns and models of patterns that are indicative of a plurality of camera systems; and comparing unknown image data to ones of the plurality of patterns so as to identify a corresponding one of the camera systems as a source of the unknown image data.

3. A method as claimed in claim 1, wherein said patterns include patterns of pixel information of a digital image.

4. A method as claimed in claim 3, wherein said pixel information is pixel information of an optical scanner.

5. A method as claimed in claim 3, wherein said pixel information is pixel information of a CCD chip.

6. A method as claimed in claim 3, wherein said pixel information is dark current of a digital imaging system.

7. A method as claimed in claim 1, wherein the patterns include at least one of a feature vector and fixed pattern noise and environmental effects and temperature effects and signal processing effects.

8. A method as claimed in claim 1, wherein said comparing step generates a most likely candidate for identity.

9. A method as claimed in claim 8, wherein said comparing step generates a confidence indicator.

10. A method as claimed in claim 1, wherein said step of operating the camera system includes operating the camera system in a dark box.

11. A method for identifying a common source for image data, comprising the steps of: receiving first image data; detecting first source specific artifacts in the first image data; classifying the source specific artifacts; receiving second image data; detecting second source specific artifacts in the second image data; and comparing the first source specific artifacts of the first image data to the second source specific artifacts of the second image data to determine an existence or absence of at least substantial similarity of the first and second source specific artifacts.

12. A method as claimed in claim 11 , wherein said the second image data includes image data of a plurality of image sources and further comprising the steps of: storing the second source specific artifacts of a plurality of image sources in a database as a plurality of second source specific artifacts; and comparing the first source specific artifact to the plurality of second source specific artifacts.

13. A method as claimed in claim 11, wherein said source specific artifacts include at least one of feature vectors and fixed pattern noise.

14. A method as claimed in claim 11, wherein said step of comparing includes an identification of a most likely candidate for identity and an confidence factor.

15. A method as claimed in claim 11, wherein said first image data and said second image data are portions of a same video sequence, and said comparing step determines if a manipulation has occurred in the video sequence.

16. A system for determining a source of image data, comprising: an image data source; software stored on a computer readable media and operable on a computer to perform the steps of: receiving the image data from the data source; determining source specific artifacts in the image data; and comparing the source specific artifacts to source specific artifacts of known image data.

17. A system as claimed in claim 16, wherein said image data source is recorded media on which is stored image data.

18. A system as claimed in claim 16, wherein said image data source is a camera system.

19. A system as claimed in claim 16, wherein said source specific artifacts include at least one of vector features and temperature effects.

20. A system as claimed in claim 16, wherein said system is operable in at least one of the following modes: a verification mode, an identification mode, and a fault detection mode.