WO2010062290A1 - Multiphase watermark patterns - Google Patents

Abstract

Multiphase watermarks in accordance with embodiments of the invention include designs that alternate between predefined states. Various portions of the watermark pattern are embedded in some locations and not embedded in other locations. Extraction of multi phase watermark utilizes cumulation of differences between different phases holding portions of the watermark pattern thus diminishing the media content, emphasizing the watermark pattern and averaging background more efficiently and leading to a stronger, more robust detection; this way, non-blind extraction is mimicked without the use of the original unmarked content.

Description

MULTIPHASE WATERMARK PATTERNS

FIELD OF THE INVENTION

[0001] The present invention relates generally to identification of ownership of media objects as they are disseminated and more specifically to techniques for watermarking media objects.

BACKGROUND OF THE INVENTION

[0002] The term "media object" is often used to refer to an audio, video, graphic, multimedia, textual, 3D representations and/or interactive data file that is delivered to a user device via fixed media, the Internet or another communication network. Media objects also include streaming media files that are transferred through a networked environment and begin to play on a user device before delivery of the entire file is completed, digitally recorded music, movies, trailers, news reports, radio broadcasts and live events. Means for accessing a communication network to obtain media files include high-bandwidth connections such as 3 G wireless, cable, satellite, DSL and Tl communication lines.

[0003] Digital representation, storage, distribution, and duplication of digital media objects have become popular because digital media objects are inexpensive, easy to use, and maintain the quality of the media. However, these advantages have enabled widespread, illegal distribution and use of copyrighted material, such as unauthorized distribution of digital images, audio, and video over the Internet. Many different approaches have been presented to secure digital media objects against unauthorized use. For example, digital encryption technology is effective to prevent unauthorized interception during delivery. However, after delivery, when the content is decrypted and presented in a form that is visible or audible to humans, the content can be re-recorded and an unsecured copy can be obtained and distributed.

[0004] Marking media objects by embedding recipient information in the media can help to identify individuals that receive a media object and use the content in an unauthorized manner. Embedding ownership information in a media object can also indicate copy restriction and clarify ownership of the media object. [0005] One way of marking media objects is by adding annotations to the digital media object file format that can be read from the unmodified file and that are ignored during playback. This information is often lost, however, if the file is re-recorded or converted to another format. To achieve a robust and permanent mark in video media, visible and overlay images that display copyright information during playback have been proposed. Overlaid images are typically more robust against modification than annotations to the digital media object file and are easy to read. However, this approach can reduce the quality of the marked video, because the overlaid images interfere with the original video and diminish the quality of the viewing experience. In addition, overlaid images can be easy to identify, and remove by overwriting or cropping. [0006] Another approach for inserting information into a media object is to apply a watermark to the content of the media object. Watermarking approaches generally embed a specific watermark pattern in the media content, this pattern might be generated using the frequency domain such as DCT, Fast Fourier, or Wavelet or it might be a pattern that is expressed with a global modulation of the media over time. It can be embedded in spread spectrum fashion, as a collection of dots, a human readable graphic or in one of the many other ways used to translate information in media modification for the purpose of marking the media. For detection of the watermark, the pattern might be recognized in a transformed domain, such as wavelet, DCT, Fourier or in an image in the spatial domain in luminance or color, in image formats such as YUV, RGB, etc. The pattern might be recognized through comparison with a known or suspected pattern; it might be detected by looking for peaks or relationships of elements or might represent a human recognizable or human readable image. The interpretation of watermarking by embedding a pattern is true for many approaches of watermarking applied to media objects of video, image, audio, text and other forms of media.

[0007] Digital still images were the focus of early watermarking research and many common video watermarking approaches simply involve the application of a still image watermarking technique to each video frame. This approach can fail if the watermark cannot be read in any of the frames due to a failure in registration or destruction of relevant areas of the video frames. Furthermore, if frames are watermarked as individual images and the watermarks vary between frames, the watermarks are susceptible to an attack, because similar frames within one video can be averaged together in order to weaken the watermark. Alternatively, when each of the frames contains an identical watermark the frames can be used to analyze the structure of the watermark that is in each of the frames to understand and subsequently remove the watermark. [0008] A watermark pattern is typically embedded in order to be readable at a later stage. This reading process is also called extraction, detection, interpretation or readout. It can be performed by a human operator or by a machine algorithm, depending on the approach that is used. For extraction procedures that read information that has been combined with the media object, it is sometimes a requirement and most often an advantage if the unmarked original media object that was used to create the watermarked media object can be used in comparison. The comparison can be used to derive the watermark pattern as a difference between the unmarked media object and the marked media object that contains the watermark pattern. The derived differences can enable detection of the embedded watermark message on a shorter or smaller sample of the media object containing the watermark pattern. The approach of using the original, unmarked media object during watermark detection is also called informed or non-blind detection. The fact that non-blind detection requires the original for detection can be a limitation. Often, the original media object is not known before extraction of the watermark. In many instances, the original varies depending on video parameters such as sampling rate, aspect ratio, and resolution. Hence the title and owner of a media object are often insufficient to identify the original media object. The parameters used for encoding and distribution are also required for extracting watermark information. Another consideration for obtaining the original unmarked content is that it might be well protected and not easily obtained in an unsecured, directly accessible, unmarked format.

[0009] Comparing unmarked content to the obtained marked content can be technically challenging. The marked content has often undergone some significant transformations such as compression, change of aspect ratio, scaling, cropping, other geometric transformations, re- recording with a camcorder, and/or change in luminance or color distribution. Comparisons will not only yield the difference introduced through the watermark pattern, which is the intended result of the comparison, it will also yield all differences introduced by the variations mentioned above. The challenge of matching a distorted and misplaced watermark pattern to the expected pattern location is also called synchronization. Synchronization is required for every domain of the media object. In the case of video, synchronization can involve matching to the original in the spatial, time and color domains.

[0010] Several watermarking approaches use complete negative and positive patterns that are used in different spatial areas of the frames, so called fields. The recognition of this information requires the separation into fields with classification in fields with positive and negative patterns. This is not regularly possible e.g. after modifications that change the size of the content, change the frame rate, convert it from interlaced to progressive content, or modify it geometrically through operations like rotation. The embedding of negative and positive information also involves the detection mechanism adapting to the fact that the information is embedded in a varying manner.

SUMMARY OF INVENTION

[0011] Systems and methods are described for distributing watermark patterns across multiple media units of a media object. One embodiment of the invention includes dividing a watermark pattern into a plurality of sub patterns wherein at least one of the sub patterns is missing a portion of the complete watermark pattern and combining each sub pattern with a different media unit of the media object in a repeated embedding cycle, thereby creating a modified media object with modified media units. In addition, the complete watermark pattern can be recovered from the modified media object by observing differences between modified media units in which different sub patterns are embedded.

[0012] In a further embodiment, the media object is digital audio and the media units are audio intervals.

[0013] In another embodiment, the media object is a video sequence and the media units are video intervals.

[0014] In a still further embodiment, the video interval corresponds to a single frame of video.

[0015] In still another embodiment, the watermark pattern includes human recognizable symbols.

[0016] In a yet further embodiment, the watermark pattern is divided into a plurality of sub patterns, where each sub pattern resembles film grain and at least a portion of the watermark pattern is represented by differences in the film grain between different sub patterns, and the sub patterns are combined within the frames of the video sequence so as to appear as film grain to a human observer.

[0017] In yet another embodiment, combining each sub pattern with a different media unit further comprises modifying the sub pattern according to the content of the media unit using a perceptual model before combination.

[0018] In a further embodiment again, combining each sub pattern with a different media unit of the media object in a repeated embedding cycle, thereby creating a modified media object with modified media units further includes determining whether the content in a sequence of consecutive media units contains a level of variation below a predetermined threshold, when the level of variation is below the predetermined threshold, combining each sub pattern with a different media unit in the sequence of consecutive media units in a repeated embedding cycle, and when the level of variation is above the predetermined threshold, embedding the watermark pattern in the sequence of consecutive media units using an alternative embedding process.

[0019] In another embodiment again, the watermark pattern is represented in a frequency domain.

[0020] In a further additional embodiment, creating a plurality of sub patterns using the watermark pattern includes creating at least one complimentary pair of sub patterns, wherein each sub pattern in the complimentary pair includes a different portion of the watermark pattern and the pair can be combined to recreate the complete watermark pattern.

[0021] In another additional embodiment, the embedding cycle involves embedding different sub-patterns in media units separated by a predetermined temporal distance.

[0022] In a still yet further embodiment, the predetermined temporal distance varies in a predetermined way as the embedding cycle repeats.

[0023] In still yet another embodiment, the media object is a video sequence and the media units are individual frames of video, and the predetermined temporal distance between media units is chosen to minimize flicker in the modified video sequence.

[0024] In a still further embodiment again, the embedding cycle is a rolling cycle where creating a plurality of sub patterns using the watermark pattern further includes creating a first sub pattern using a first portion of the watermark pattern, creating additional sub patterns by combining an additional portion of the watermark pattern with the previous combination until a sub pattern is created that includes the entire watermark pattern, creating additional sub patterns by removing a portion of the watermark pattern from the combination in the order the portions of the watermark pattern were originally added to the combination until a sub pattern is obtained that does not contain any portion of the watermark pattern and, in addition, the sub patterns form complimentary pairs, where each complimentary pair includes a different portion of the watermark pattern, and the pair can be combined to recreate the complete watermark pattern. Furthermore, combining each sub pattern with a different media unit in a repeated embedding cycle includes combining each of the sequence of sub patterns with each media unit in an order corresponding to the order in which portions of the watermark pattern were added and removed to create the sub patterns.

[0025] 15. The method of claim 1, wherein the separation of the watermark pattern into sub patterns varies between embedding cycles and the sub patterns can be chosen on pseudorandom basis.

[0026] Still another embodiment again includes dividing a watermark pattern into a positive pattern including the complete watermark and a negative pattern including an inverse of the complete watermark, such that the positive and negative patterns cancel when combined, and combining the positive and negative patterns with different frames of the video sequence in a repeated embedding cycle, thereby creating a modified video sequence with the modified frames. In addition, the complete watermark pattern can be recovered from the modified video sequence by observing differences between a modified frame in which a positive pattern is embedded and a modified frame in which the negative pattern is embedded.

[0027] A still further additional embodiment includes selecting at least one pair of media units from the media object, where each of the pair is likely to include a different sub pattern, subtracting one media unit in the pair from the other media unit to create a difference media unit in which the sub patterns are emphasized relative to the content of the media units, and estimating the watermark pattern using at least the difference media unit. [0028] In still another additional embodiment, selecting multiple pairs of media units, where the media units in each pair are likely to include different sub patterns, subtracting one media unit from the other media unit within each pair of media units to create difference media units in which the sub patterns are emphasized relative to the content of the media units, combining the difference media units to create an accumulated media unit in which the parts of the watermark pattern from each of the sub patterns is emphasized, and estimating the watermark pattern using the accumulated media unit.

[0029] In a yet further embodiment again, the absolute differences of the media units are combined to create the accumulated media unit.

[0030] In yet another embodiment again, selecting at least one pair of media units in the video sequence that is likely to include different sub patterns further comprises selecting pairs of media units that are likely to include complimentary sub patterns.

[0031] In a yet further additional embodiment, the media object is digital audio and the media units are intervals of audio.

[0032] In yet another additional embodiment, the watermark pattern is represented in a frequency domain

[0033] In a further additional embodiment again, the media object is a video sequence and the media units are individual frames of video.

[0034] In another additional embodiment again, the watermark pattern is a human recognizable image.

[0035] In a still yet further embodiment again, the marked media object contains consecutive media units in which the watermark pattern is embedded using an embedding process that does not utilize sub patterns and the method of recovering the watermark pattern includes determining whether the content in a sequence of consecutive media units contains a level of variation below a predetermined threshold, when the level of variation is below the predetermined threshold, recovering information concerning the watermark pattern by selecting multiple pairs of media units from the sequence of consecutive media units, where the media units in each pair are likely to include different sub patterns, subtracting one media unit from the other media unit within each pair of media units to create difference media units in which the sub patterns are emphasized relative to the content of the media units, combining the difference media units to create an accumulated media unit in which the parts of the watermark pattern from each of the sub patterns is emphasized, and estimating the watermark pattern using the accumulated media unit, in addition, when the level of variation is above the predetermined threshold, recovering information concerning the watermark pattern from the sequence of consecutive media units using an alternative extraction process, and combining the recovered information to estimate the watermark pattern.

[0036] In still yet another embodiment again, the media object is a video sequence, the media units are individual frames of video, and the watermark pattern is a human recognizable image, and the alternative extraction process comprises combining frames of video in a predetermined manner that results in an accumulated image in which the parts of the watermark pattern present in the frames of video are emphasized.

[0037] In a still yet further additional embodiment, selecting at least one pair of media units includes selecting pairs of media units separated by an initial temporal distance and generating an initial accumulated media unit over a first portion of the media object, generating multiple accumulated media units by selecting pairs of media units from a second portion of the media object, wherein each of the multiple accumulated media units is generated by selecting a different temporal distances between the pairs of media units selected from the second portion of the media object, and comparing each of the multiple accumulated media units with the initial accumulated media unit to select the temporal distance that yields the greatest continuity of results over the second portion of the media object. In addition, combining the difference media units to create an accumulated media unit in which the parts of the watermark pattern from each of the sub patterns is emphasized includes combining the initial accumulated media unit with the accumulated media unit generated with the temporal distance that yields the greatest continuity to create an accumulated media unit in which the parts of the watermark pattern from each of the sub patterns is emphasized.

[0038] In still yet another further embodiment, selecting at least one pair of media units includes selecting pairs of media units separated by a selected temporal distance and generating an initial accumulated media unit over a first portion of the media object, generating multiple accumulated media units by selecting pairs of media units separated by the selected temporal distance from within a second portion of the media object, where each of the multiple accumulated media units is generated by selecting a different start time from which to commence selecting the pairs of media units from within the second portion of the media object, and comparing each of the multiple accumulated media units with the initial accumulated media unit to select the start time that yields the greatest continuity of results over the second portion of the media object. In addition, combining the difference media units to create an accumulated media unit in which the parts of the watermark pattern from each of the sub patterns is emphasized includes combining the initial accumulated media unit with the accumulated media unit generated with the start time that yields the greatest continuity to create an accumulated media unit in which the parts of the watermark pattern from each of the sub patterns is emphasized. [0039] In a still yet further additional embodiment again, the media object is a video sequence and the comparison is performed with respect to results accumulated over portions of the video sequence corresponding to three seconds of video.

[0040] In still yet another additional embodiment again, selecting at least one pair of media units from the media object further comprises selecting a pair of media units separated by a predetermined temporal distance.

BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The drawings are not necessarily to scale, emphasis instead generally being placed upon illustrating the principles of the invention.

[0042] FIG. 1 is a flow chart showing a process for embedding a multiphase watermark in accordance with an embodiment of the invention.

[0043] FIG. 1 a illustrates a series of unmarked video frames.

[0044] FIG. Ib illustrates a complete watermark pattern.

[0045] FIG. Ic illustrates an example of a watermark sub pattern generated in accordance with an embodiment of the invention.

[0046] FIG. 1 d illustrates a mutually exclusive watermark sub pattern that is complementary to the watermark pattern in Fig. Ic generated in accordance with an embodiment of the invention.

[0047] FIG. Ie illustrates the video frames in Fig Ia that contain the watermark patterns illustrated in Fig Ib, divided into watermark sub patterns as illustrated in Fig. Ic and Id and applied to the video frames in accordance with an embodiment of the invention.

[0048] FIG. 2 is a flow chart showing a process for extracting a multiphase watermark in accordance with an embodiment of the invention.

[0049] FIG. 3 is a conceptual illustration of a process for extracting a two phase watermark in accordance with an embodiment of the invention. [0050] FIG. 4 is a conceptual illustration of a process for extracting a rolling phase watermark in accordance with an embodiment of the invention.

[0051] FIG. 5 is a conceptual illustration of a process for extracting a multiphase watermark that includes positive and negative sub patterns in accordance with an embodiment of the invention.

[0052] FIG. 6 is a conceptual illustration of a process for creating a multiphase watermark that has the appearance of film grain and extracting a watermark pattern from the film grain in accordance with an embodiment of the invention.

DETAILED DESCRIPTION

[0053] Turning now to the drawings, systems and methods for embedding multiple phase watermark patterns into media objects in accordance with embodiments of the invention are described. In many embodiments, a watermark pattern is divided into several sub patterns derived from the watermark pattern and the sub patterns are then repeatedly embedded in the media object in accordance with a known cycle. A watermark pattern, as depicted in FIG. Ib can be divided into two non-overlapping, mutually exclusive sub patterns (i.e. each sub pattern is derived from separate and distinct portions of the watermark pattern and the sub patterns combine to form the complete watermark pattern) as is shown in FIGS. Ic and Id. FIG. Ie shows marked frames with the sub patterns applied to video frames of FIG. 1 a. [0054] The repeated embedding of the watermark sub patterns distributes information throughout the media object that can be used to reconstruct the complete watermark pattern. The sub patterns are recovered by exploiting the fact that portions of the media object that are adjacent in time are typically similar. The sub patterns are accentuated by subtracting portions of the media object that are likely to contain similar content and different sub patterns thereby decreasing the signal strength of the content in relationship to the signal strength of the watermark pattern.

[0055] In many embodiments, a sequential cycle is used repeatedly when embedding sub patterns into a media object so that the distance between corresponding sub patterns is constant regardless of the starting point used for extraction and the watermark pattern can be recovered without knowledge of the temporal embedding locations of the watermark sub patterns. When sequential embedding is used, each sub pattern can be assigned to a phase in an embedding cycle. The temporal distance between corresponding phases is typically chosen to be small such that the content in the media object of corresponding phases is likely similar because the temporal distance between them is small. On the other hand, when phases are spaced too close to each other and the portions of the media object in which they are embedded are almost identical, then the variations introduced by the two embedded sub patterns are more likely to become noticeable to a human observer.

[0056] In a number of embodiments, a rolling cycle is used when embedding sub patterns. A rolling cycle involves starting with a first sub pattern in a first embedding location and at each successive embedding location combining an additional portion of the watermark pattern with the previous combination until all of the sub patterns are combined and the entire watermark pattern is embedded at an embedding location. When the full watermark pattern is embedded at an embedding location, the rolling cycle continues by removing a portion of the watermark pattern from the combination at each subsequent embedding location in the order the portions of the watermark pattern were originally added until a sub pattern is obtained that does not contain any portion of the watermark pattern. At which point, the rolling cycle repeats. The advantage here is that the combination of phases during extraction can be slightly off and the difference still yields the majority of the complete watermark pattern. Furthermore, the difference in the pattern between phases is small resulting in less visible distortion.

[0057] Multiphase watermarks in accordance with many embodiments of the invention can be used in conjunction with a variety of techniques for generating watermark patterns. The distribution of the watermark pattern over several phases does not generally prevent the readout of the watermark pattern using common detection approaches such that the watermark can still be read, at least in part, even if the multi phase distribution is not taken into account. Allowing for readout of the same pattern using different extraction techniques increases the robustness of the watermark embedding process to attacks designed to frustrate specific extraction processes. [0058] Separation of a watermark pattern into sub patterns, in accordance with embodiments of the invention, can provide security advantages. The multiphase watermark can be chosen and embedded so that the number of phases and cycle length must be known in order to be able to recover a complete watermark pattern. Access to this information can be restricted by keeping the number of phases and length of the cycle secret and additional security can be introduced by variation of these values in regular or pseudo random intervals that are difficult to guess and, thereby, further secure access to the watermark pattern.

[0059] Another security advantage is that a multiphase watermark pattern in accordance with embodiments of the invention can be extracted in spite of the combination of the marked video with a constant pattern that might be present or introduced to interfere with a watermark pattern. Examples of such patterns include a noise pattern introduced to hinder the watermark readout, dust or spots on a TV screen or a moire pattern that can occur when re-recording content with a camera from the TV. While these patterns are typically not intended, they can interfere with the recognition of a number of prior art watermark patterns in a manner similar to the interference introduced by intentionally adding a noise signal. By using the differences between portions of the media object during extraction of a multiphase watermark, a constant noise that might hinder the extraction of constant watermark patterns does not significantly interfere with the extraction process.

[0060] In a number of embodiments, the media object is a video sequence and the watermark pattern is an image. Sub patterns are derived from the watermark image by dividing the watermark image into separate and distinct watermark image portions that can be recombined to form the complete watermark pattern. Each sub pattern is then repeatedly and imperceptibly embedded within frames from the video sequence. The selection of frames in which to embed the sub patterns can influence whether the embedded sub patterns are perceptible to a human observer. During extraction, frames can be subtracted from each other to emphasize a sub pattern present in one of the frames, or the sub patterns present in both frames. When the content of the frames is similar, which is typically the case when the embedding cycle is relatively short, the subtraction suppresses the media content and emphasizes the embedded sub pattern(s). In several embodiments, the two frames that are subtracted from each other are chosen to contain sub patterns that are complimentary so that the difference frame contains the entire watermark image. Readout of the complete watermark can be improved by summation or averaging of the difference frames.

[0061] In several embodiments, the media object is an audio sequence and the embedding cycle includes phases defined as time intervals of the audio content. Similar intervals that are closely co-located in time will yield similar frequency content and frequencies in the current interval can be used to estimate the frequency in the neighboring intervals. When a watermark pattern is embedded as additional frequencies, which are present in one interval and not in the next, the content of one audio interval can be used to estimate another neighboring interval in order to decrease the signal of the underlying carrier for the watermark pattern and thereby emphasize the watermark pattern. Multiphase watermarks, embedding processes and extraction processes are discussed in detail below.

Embedding Multiphase Watermarks in Media Objects

[0062] A process for embedding a multiphase watermark into a media object in accordance with an embodiment of the invention is illustrated in FIG. 1. The process (10) includes creating (12) sub patterns from a source watermark, where the sub patters can each contain different portions of the watermark. The sub patterns are then rendered (14) in a form that enables combination of the sub patterns with the media object in which the multiphase watermark is to be embedded. The process is completed by combining (16) the rendered sub patterns with the media object in accordance with a predetermined embedding cycle. The embedding cycle can be any predetermined process for repeatedly embedding the sub patterns and/or combinations of sub patterns in the media object. In many instances, the choice of embedding cycle used is influenced by the availability of the original media object during the extraction process. Complex embedding cycles typically require the availability of the original media. In a number of embodiments, the predictable nature of sequential embedding cycles enables the extraction of a multiphase watermark without access to the original media object. Processes for embedding and extracting multiphase watermarks are discussed further below.

Embedding Multiphase Watermark Patterns in Video

[0063] A process in accordance with an embodiment of the invention for embedding a multiphase watermark derived from a watermark pattern in a video sequence is illustrated in FIGS. Ia - Ie. A media object including a sequence of video frames is illustrated in FIG. Ia. The media object includes a video sequence HOa, which is illustrated as a series of complete frames. In other embodiments, the video sequence can be stored in the media object as compressed video. The media object may also include timing information concerning the time at which frames of video are rendered for display during the rendering of the video sequence. In the illustrated embodiment, the timing information is conceptually illustrated as a time line 155a. As will be discussed further below, timing information from the original media object can be important to the extraction of multiphase watermarks when the original media object is not available.

[0064] A watermark pattern containing information for embedding in the video sequence 110a shown in FIG. Ia in accordance with an embodiment of the invention is illustrated in FIG. Ib. The complete watermark pattern 101b is a single image that includes information. The use of the information contained within a watermark pattern in various applications is described in detail below. In the illustrated embodiment, the pattern consists of two rows of characters ABC and DEF although any pattern representing encoded information could be used as the basis of the multiphase watermark. In some embodiments, the pattern includes human recognizable components such as characters, numbers or bar codes. In many embodiments, the watermark is a noise like signal from which useful information can be decoded.

[0065] As discussed above, the creation of a multiphase watermark involves dividing a watermark into sub patterns. In the case of a watermark that is an image, the sub patterns can be created by dividing the image into regions. In the process illustrated in FIGS. Ia - Ie, the watermark image is divided in half along a horizontal center line of the watermark image so that each image is mutually exclusive. In the context of multiphase watermark sub patterns, the term 'mutually exclusive' refers to the property of the sub patterns that no two sub patterns include the same piece of information from the watermark. The first sub pattern image 102c is shown in FIG. Ic and includes the characters ABC from the first row of the watermark image 101b. The second sub pattern image 103d is shown in FIG. Id and includes the characters DEF from the second row of the watermark image 101b. Combining the first and second sub pattern images creates the full watermark image. In other embodiments, the watermark image can be divided into multiple regular and/or irregular sub patterns consisting of regions of the watermark. In a number of embodiments, pairs of mutually exclusive sub patterns are created such that for each sub pattern in an embedding cycle there is a corresponding complimentary sub pattern. The sub pattern and its corresponding complimentary sub pattern can be combined to obtain the complete watermark pattern. [0066] In many embodiments, embedding sub patterns instead of the complete watermark pattern can improve the processing time required to modify a portion of a media object in order to embed the complete watermark pattern (a video frame for digital video). [0067] The reduction in modification can also have a positive impact on invisibility. When fewer modifications are applied to a media object, the modifications are typically less perceptible.

[0068] A multiphase watermark image imperceptibly embedded into a video sequence in accordance with an embodiment of the invention is illustrated in FIG. Ie. The first and second sub patterns are embedded in the video sequence 1 1Oe in two phases, where each phase includes three frames of the video sequence, to form a repeating cycle of six frames. [0069] The embedding cycle that resulted in the marked sequence shown in FIG. 1 e includes two phases. In many embodiments the embedding process includes N phases. Unlike the rolling phase approach described below, each phase holds a distinct mutually exclusive portion of the watermark and no two phases hold a common portion of the watermark. For larger values of N, the sub patterns are smaller and offer improved invisibility of the watermark. In addition, less processing power is required during application of the mark. The extraction of an N phase watermark is the result of accumulation of the differences between all different phases. No two phases yield the complete watermark pattern for values of N greater than 2, where the embedded sub patters are mutually exclusive. Another advantage during extraction is that the corresponding phase does not have to be known as the content is combined, because for N > 2 there is a high probability that the combined content is of different phases. For example, when N = 4 each phase consists of a different combination of sub patterns and include ¹A of the complete watermark pattern. If during extraction, two random frames are chosen, there is a probability of 75% that they do belong to different phases and, in that case, the resulting difference contains at least 50% of the complete watermark pattern, since each of the frames contributes at least 25% of the watermark pattern. Extraction of multiphase watermarks and different multiphase watermark embedding cycles, are discussed further below.

[0070] The anti phase distribution illustrated in Fig. 5, includes negative and positive representations of the pattern. It is created by either embedding positive and negative watermark patterns in the cycle or by toggling between addition and subtraction of regions of the watermark pattern from the media object. This is typically done in two opposite phases per cycle, but may also be combined with multiphase and rolling phase approaches. When subtracting the two during extraction, the effect of suppressing the video content and highlighting the watermark pattern is strengthened at the expense of requiring the knowledge or precise estimate of all positive and all negative phases. FIG. 5 is discussed further below.

Extraction of a Multiphase Watermark

[0071] Processes for extracting multiphase watermarks from media objects can be generalized as subtracting portions of the media object from different phases of the embedding cycle to suppress media content and combining the result of the subtractions until the complete watermark is recognizable. The generalized process can be better understood by considering individual extraction processes in accordance with embodiments of the invention. [0072] A process for extracting a watermark image from a video sequence in which a multiphase watermark is imperceptibly embedded in accordance with an embodiment of the invention is illustrated in FIG. 2. The process 20 commences by digitizing (22) the marked video sequence. In the event that the marked video sequence is received in a digitized form, then digitization is not necessary (although transcoding may be necessary to convert the digitized video into an appropriate input format). A first frame can then be selected (24) from the marked video sequence and a second frame from a different, corresponding phase of the embedding cycles is also selected (26). The position of the frame in the different corresponding phases is estimated, taking into account parameters like frame rate, original embedding cycle and phase, and/or the original content for comparison. The content in the two selected frames is suppressed by subtracting (28) one frame from the other. The resulting difference between the frames emphasizes the difference between the frames. The difference is composed of the difference of the multiphase watermark sub patterns embedded in the two frames, and the difference between the content within the frames. When frames are close to each other in time, the difference between the content is typically small. However, in most motion pictures the frames will differ and while the difference of nearby frames is typically smaller than the difference between unrelated images, the difference might not be strong enough to reveal the complete watermarking pattern. For this reason, and because the watermark pattern can be degraded through video processing and/or have been embedded weakly in the frames in order to keep the sub patterns imperceptible, the resulting differences are combined to further emphasize the complete watermark image and further suppress the content, which is typically uncorrelated over time and averages out. Therefore, the process repeatedly selects and subtracts pairs of frames and combines (30) the difference frames until a determination (32) is made that the watermark image is recognizable from the combined difference frames. Although a determination need not be performed at this point and the process can simply be applied to the entire marked sequence. [0073] Performance of extraction processes in accordance with embodiments of the invention is improved by reducing instances in which frames including the same sub pattern are subtracted (i.e., frames from the same phase or from a similar phase in a multiphase approach). A subtraction involving the same sub pattern results in the loss of the portion of the watermark image embedded in both frames and increases the number of difference frames combined to extract the watermark image. In many embodiments, the frames that are combined are chosen on the basis that the sub patterns in each of the frames are complementary. Subtracting frames including complementary sub patterns can create a difference frame including a complete watermark image. Therefore, each subtraction yields the maximum amount of information concerning the watermark image. When the sub patterns are not complementary, multiple difference frames are required to obtain the complete watermark image.

[0074] Selection of frames containing appropriate sub patterns during an extraction process in accordance with an embodiment of the invention involves knowledge of or an estimate of the phase length used in the embedding of the sub patterns. The simplest manner in which to estimate the phase length used in the embedding process is to have access to the original content and information concerning the embedding process. However, the original content is often not available during the extraction process. In many embodiments, frames are chosen during the extraction process such that their distance is a multiple of the estimated embedding phase length. It is noted that the length of the phase for digital media can be measured in samples (i.e. frames for digital video), or in units of time (i.e. milliseconds for digital video). Estimating the phase length using a measurement of time instead of a measurement of samples can make the extraction process more robust to voluntary or involuntary degradation of the watermark. One common technique that is used to make a watermark unrecoverable is to re-encode a video sequence at a different frame rate to the original frame rate or to resample the video. The re- encoding process changes the number of frames and both the re-encoding and resampling processes can involve blending information from two or more frames from the original video sequence. However, the re-encoding and resampling process typically attempts to preserve the timing of the video sequence. Therefore, use of units of time instead of samples to estimate phase enables identification of portions of a media object that are likely to contain sub pattern information in circumstances where the media object has been modified.

[0075] When subtracting frames, the difference operation is applied to the embedding domain; other frame content can be ignored. In many embodiments, the multiphase watermarking pattern is embedded in the luminance domain of the video sequence such that the difference of the frames is the difference of the luma values for each pixel of the current processed frame with the selected frame, while color information is not used. In many embodiments, other embedding domains are used and the extraction process is modified accordingly.

[0076] When the subtraction process involves subtraction of a first frame containing a sub pattern from a second frame containing a sub pattern, the sub patterns are represented as both positive and negative values. Assuming that the content is identical in both frames, the sub pattern from the second frame will be represented in the difference frame as positive values and the sub pattern from the first frame will be represented as negative values. Combining difference frames that include some information concerning the watermark patterns expressed as positive values and some information concerning the watermark patterns expressed as negative values will not necessarily result in the watermark information being emphasized. Therefore, many embodiments of the invention use the absolute value of the difference frame during the combination process. When the absolute value of the difference is used, the resulting pattern contains a positive watermark pattern only and the result in respect to the watermark pattern is independent of the order in which the phases are combined. Consequently, no phase alignment or time synchronization is necessary.

[0077] When difference frames are combined using absolute values, the pattern can be identified even if the embedding cycle is not precisely known or content has been added or removed to the media object that disrupts the phases. This and other benefits of using absolute values when combining difference frames can be better understood with reference to an example. Consider two pixels pi, at position o2 and p2 at position o2 that are part of a marked frame fl . pi is part of the watermark pattern, p2 is not. The following frame f2 does not contain a watermark pattern in neither ol nor o2. For simplification all unmarked pixels of both frames including at position o2 and ol and o2 in f2 have a value of 10. While this is not generally the case, the values are likely to be similar as explained above. All pixels that are part of the slightly darker watermark pattern have a value of 7. When subtracting the pixel at position of ol in f2 from pi, the value for pi will be vl^a: 7-10 = -3 and the value for v2^a will be 10 - 10 = 0. If the phases are re-ordered and confused elsewhere in the media object such that the marked frame pi is subtracted from the unmarked frame, the resulting difference values are vl^b: 10-7 = 3 and v2 : 10 - 10 = 0. If the difference frames are directly combined, the difference values vl^a and vl will neutralize each other and the resulting pattern is not recognizable. To remedy this, the values can be turned into absolute values such that the resulting value for the marked pixel in both difference frames is 3 and the positive watermark pattern will be recognizable. If the location of the phases is known throughout the marked content, then the absolute value calculation need not be applied to more rapidly average the video content.

[0078] Multi phase watermark extraction emulates (or mimics) non-blind extraction methods without the necessity of having unmarked original content at the time of extraction. [0079] In other embodiments, extraction of a multi phase watermark is based on combining difference frames without taking the absolute value of the difference frame. Compared to extraction processes that use absolute values, using the values of the difference frames allows for stronger extraction results by more rapidly averaging the content within the difference frames. However, the extraction process is dependant upon time synchronization because for some phases the computed watermark pattern is positive and is negative for other phases. To avoid this misalignment, negative values may be set to 0, which effectively eliminates these watermark signals, but that might be preferable to inversion or cancellation of the previous extraction results. In the case of a rolling phase watermark (see discussion below), one solution to this impediment is to repeat watermark extraction for each complimentary phase combination independently and combine the end results, such that the pattern can be recognized in extraction from each phase even if some regions are positive others are negative. If the information is represented with symbols as described in Patent 7,430,302 entitled "Covert and Robust Mark for Media Identification", the symbols might be light and dark but still readable either way. The same symbol or region might be darker in one result and lighter in another. [0080] Multiphase watermark extraction based on combining difference frames without taking the absolute value of the frames is susceptible to frame drops and other discontinuities of time domain. In these instances, the extraction of a specific phase will jump phase reducing the extraction results. Phase jumps during no absolute value extraction and the ensuing weakening of the extracted watermark pattern can be prevented through performing extraction from various portions of the video content using the entire range of phase shifts and correlating these partial extraction results to maintain correct phase throughout extraction.

[0081] A number of general principles for the extraction of a variety of different types of multiphase watermarks are described above. Specific examples of multiphase watermarks and techniques for extracting the multiphase watermarks in accordance with embodiments of the invention are discussed below.

Extraction of Multiphase Watermarks using Complimentary Sub Patterns [0082] A process for extracting the multiphase watermark embedded in the video sequence HOe of FIG. Ie is conceptually illustrated in FIG. 3. The video sequence HOe includes a multiphase watermark having two phases, where each phase includes embedding one of a pair of complimentary sub patterns in three successive frames of the video sequence to form a repeating cycle of six frames. Extraction of the watermark involves selection of pairs of frames from the different phases. The phases are 3 frames long. Therefore, selection of frames separated by a time equivalent to the temporal distance between 3 frames in the original video sequence is likely to result in selection of a pair of frames from the different phases. The start or end of the phase does not have to be known, it is sufficient to know the length of the phase and any frame with the required time distance will be in a different phase. In the illustrated embodiment, the arrows 120 indicate the temporal distance used to select pairs of frames from the marked video sequence. Two copies of the marked video sequence are shown in FIG. 3 to help conceptualize the selection of pairs of frames. The subtraction results in the sequence of difference frames 130. Due to the complimentary nature of the sub patterns in the two phases, each difference frame includes information from the complete watermark image. If the selected frames are identical and the process is lossless, the difference between the frames yields nothing but the complete watermark pattern. In reality, however, the frames are likely to differ slightly and video processing during embedding introduces loss to the embedded sub patterns. By selecting frames that are near to each other, the likelihood of the frames being similar is high and the content of the video signal can be significantly reduced in comparison to the watermark pattern. Combining the difference frames further emphasizes the complete watermark image. The complete watermark image is the same in each frame and the portion of the difference frame related to the underlying content of the video sequence typically varies over time and is not reinforced by the combination process. Additional schemes for embedding and extracting multiphase watermarks are discussed below.

Rolling Multiphase Watermarks

[0083] Extraction of a rolling multiphase watermark from a marked video sequence in accordance with an embodiment of the invention is illustrated in FIG. 4. In this example, a multiphase watermark is created by splitting a complete watermark image into N parts, which are then combined to create 2N different sub pattern combinations containing N mutually exclusive regions. The sub pattern combinations are created by adding the N regions in a cumulative fashion to each proceeding sub pattern combination until the complete watermark pattern is represented, and the next N sub patterns are created by subtracting the regions in the same order as they were added (i.e. the first region added to the combination is the first region subtracted from the combination). In other words, the parts are added together, until the resulting sub pattern is the complete watermark pattern and than subtracted. In the embodiment illustrated in FIG. 4, a multiphase watermark has been embedded in the frames of a digital video sequence 210 that uses three sub patterns derived from the watermark image shown in FIG. Ib combined to form six different sub pattern combinations. The complete watermark pattern is shown in time interval 1 of 260. The complete watermark pattern is represented by the letters ABCDEF. The watermark pattern is divided into three regions where each region includes one column of the complete watermark pattern: A D, B E, C F. The three sub patterns are combined in the following 6 sub pattern combinations: ABCDEF, AB DE, A D, , C F, BC EF. The sub pattern combinations are designed such that the subtraction of frames N inter-frame time intervals apart (three frames apart in the illustrated embodiment) results in the creation of difference frames containing information concerning the entire watermark pattern. [0084] A single phase can be selected out of a rolling phase watermarking scheme for extraction. In the embodiment illustrated in FIG. 4, for which the cycle length is 6 frames, an extraction sequence would start with a randomly chosen frame N that is subtracted from frame N+3 the result is combined with the difference frames of N+6,N+9 and so on. One advantage of this approach is that the difference of two different phases contains most of the total watermark pattern, even if the distance between two distinct phases is not estimated precisely but is wrong by a small amount. For example, the combination of phase 2 containing the sub pattern AB DE with the correct corresponding phase 5 results in the complete watermark pattern ABCDEF. If the combination is instead performed with the nearby but different phase such as 6, the difference will yield the pattern A CD F which is not the complete watermark pattern but still contains the majority of the information of the complete watermark pattern.

Positive/Negative Multiphase Watermarks

[0085] Extraction of a positive/negative multiphase watermark, also called an anti phase watermark, embedded in a video sequence in accordance with an embodiment of the invention is illustrated in FIG. 5. A positive/negative multiphase watermark is embedded using an embedding cycle that includes two phases, where each phase contains the complete watermark pattern, one positive 170 and one negative 180, situated in two different frames and therefore different locations in time being displayed at distinct time intervals. The result of the differences between the two phases is a strong complete watermark pattern. The strength of the watermark pattern is typically greater than in circumstances where only differences resulting form the presence or absence of a watermark pattern can be used to emphasize the complete watermark image. In addition, the introduced modification is neutral when accumulated over several phases. This makes the marking more secure against analysis by a potential attacker when integrating phases by summation or averaging; however it also prevents interpretation of the watermark pattern in this integrated result, such that the multi phase nature of the signal has to be taken into account during extraction.

Multiphase Watermarks that Simulate Film Grain

[0086] Film grain is the noise like pattern that appears on video recorded on analog film.

Film grain is composed of high frequencies and is a random pattern with little redundancy in space or time. It is therefore difficult to compress and when compressed with lossy compression the film grain will be removed or weakened. Film grain also does not occur when the original content is captured digitally without using analog film. Although film grain is a noise like distortion that is not originally planned for but rather a byproduct of the recording technique, it is seen as a desirable feature by many viewers that are used to film grain or perceive it as an indicator of high video quality. Techniques have been proposed to simulate film grain and to add it to digital video content in order to improve the perceived quality of the video. This is typically done by overlaying noise like images that vary in space and over time. The characteristics of the film grain can be parameterized and can be transmitted along with the compressed video to be reproduced in the playback device. Specifications for film grain parameters can be found in compression format specifications such as H.264. [0087] In many embodiments, multiphase watermarks are used in combination with film grain, where the film grain pattern is chosen to still appear randomly distributed in space and time and yet changes in a specific way that can be used to embed a distributed watermark pattern. One simple way of distribution is shown in FIG. 6. The watermarking pattern to be embedded is shown in a watermark image 510 with some areas lighter (525) than others (520). The film grain pattern for an arbitrary frame is shown in a first sub pattern frame 540. The resolution is coarse to visualize the concept and the pattern can be taken to be a cropped fraction of an entire frame. This pattern is random and designed according to the desired appearance of the film grain. The second sub pattern frame 545 also appears random to the observer however, it is designed to be the opposite of the pattern 540 in areas that are lighter (525) in the watermark image 510. In the areas that are darker (520) the pattern in the second sub pattern frame 545 is random. To better visualize this concept the top left corner of each image bounded by a pair of imaginary lines 500 and 505 has been shaded and another pair of imaginary lines 530 and 535 used to connect corresponding locations in each frame for lighter and darker areas respectively. If the two noise-like film grain sub pattern frames 540 and 545 are subtracted the resulting image is the difference frame 550 that contains a pattern 560. The pattern can be recognized as PATT (560) because it is consistently white. The difference of the two film grain patterns results in maximum difference. The area outside the pattern (565) contains random noise from the difference of two random patterns. The embodiment illustrated in FIG. 6 shows that the sub patterns can be designed to appear random while their combination reveals a message. In many embodiments, the sub patterns do not vary between only two sub pattern frames, as shown, but between several sub pattern frames to allow for more than two phases and for a rolling phase approach. As a generalization, each film grain overlay acts as a sub pattern that can be combined to obtain the complete watermark. However, individual sub patterns appear noise-like. [0088] An advantage of using sub patterns having the appearance of film grain is that they may be applied in the same manner as film grain overlays and therefore can be applied in a more efficient manner. The application of film grain already requires the manipulation of all pixels. A further advantage is the fact that the film grain is designed to be visually apparent and does not have to be imperceptible as is the case with typical watermark patterns. The fact that the sub pattern can be embedded with enough strength to be perceived as film grain results in a watermark that has a higher robustness compared to watermarks that are imperceptible. As a consequence, the multiphase watermark can also be applied without use of a perceptual model. [0089] In order to apply several noise-like patterns in an efficient manner, the film grain sub patterns can be created and stored in memory during an initialization of the multiphase watermark patterns and the memory used in a circular manner during application of the film grain pattern. Manipulations may be applied if processing time allows for increasing the visual appearance of actual film grain noise.

[0090] In many embodiments, the appearance of the combined image is further enhanced by increasing the similarity between the pattern outside the graphic in order to increase performance during reconstruction of the pattern and to make the content outside the pattern less noisy. To further increase robustness and security, the area the is part of the watermark pattern (foreground) and the area outside the pattern (background) can be embedded using different cycle lengths, such that it is more likely during extraction to match the correct cycle for either foreground or background and only one cycle length needs to be approximated in order to retrieve the mark. If both cycle lengths are estimated and the foreground and background are extracted, the results can be combined for a stronger final result. For this approach it is preferred that the foreground and background cycle length have few common multiples such that the extraction of the foreground does not interfere with the background extraction and vice versa. [0091] In further embodiments, the inside of the pattern is determined so as not to be the exact opposite of a complementary sub pattern and/or to increase the random noise-like appearance for the observer and/or a machine attempting to locate the watermark.

Additional Variations to Application Parameters

[0092] As is illustrated by the many embodiments discussed above, there are many different ways in which a multiphase watermark can be generated from a complete watermark pattern. Possible considerations for selecting a multiphase watermarking technique include security against analyzing the mark, invisibility and embedding speed during application of the mark.

Pseudo Random Division of Watermark Pattern into Sub Patterns

[0093] The separation of the complete watermark patterns into sub patterns can vary between cycles and be chosen on a pseudo random basis, making it harder to understand the structure and to remove the mark. The security is increased and the simplicity of the approach is maintained in that the neither the information concerning the shape and content of the complete watermark, nor the shape of the varying sub patterns is required during extraction. However, as noted above it is in the interest of a robust extraction to maintain the same cycle structure in order to be able to perform the extraction without being aware of cycle boundaries. A compromise between the complexity of extraction and the security is established by varying the cycles in irregular but rare locations such that these boundaries are not required to maintain a good extraction result yet they would be required to remove the mark in all intervals.

Pixel or Block Based Sub Patterns

[0094] Separation of watermark images into sub patterns can occur in larger regions as for example suggested in FIG. 1 , where the upper row of characters ABC and the lower row DEF are part of different phases. This however is not a requirement for the invention to be useful. The sub pattern may also be split into smaller elements such that, for example, each pixel or block of pixels of the watermark pattern is assigned to different phases in a random, checkerboard pattern, or similar pattern. Advantages of smaller units that are distributed to different phases include improved invisibility, as modifications applied through the watermark pattern are not clustered but distributed and, therefore, are less visible. Disadvantages include the ri'sk of the pattern being weakened through low pass filtering that occurs during size reduction or compression of the content and increased complexity during application of the pattern if pixels are treated individually and distributed pixels cannot be processed in clusters.

Flicker Reduction

[0095] The length of the phase used during an embedding process can be varied and depends on considerations for invisibility and robustness. A short phase length translates into a frequently changing pattern, if the different phases are applied to different frames of a video sequence. This can translate into flicker for the observer. If the variation is very fast, the flicker is less perceptible and if it is very slow it also is less perceptible as flicker, consequently there is a frequency that is more likely to result in perceptible flicker. Longer phases exhibit less visible artifacts, however, the frames that are combined during extraction are further apart and therefore less likely to exhibit a strong correlation, and consequently are less likely to cancel out the actual video content when subtracted. For a very short phase there is a risk to the strength of the signal, when frames are blurred together. This may occur during video processing such as analog recapture or compression. Longer phases are more robust against this kind of modification since it is more likely that neighboring elements are part of the same phase. In many embodiments a total phase length after which the sub patterns change, will consider this trade off. In several embodiments, an embedding cycle of about 250 milliseconds is used. In a number of embodiments, embedding cycles of between 16 and 1000 milliseconds or are used depending on the optimization of the parameters described above and the video content and display technology. In many instances, phase lengths of between 200 - 267 milliseconds result in minimal flicker visibility.

Selective Application of Multiphase Watermarks

[0096] Multiphase watermarking processes use combinations of marked frames to reduce unwanted content and emphasize the embedded watermark. Reduction occurs when the content elements in different phases are similar and the difference of the content elements is therefore small. This effect is most powerful, when the variation in the content is small. Examples include content of a slow moving or even static scene in which frames are very similar. Consequently this approach is particularly desirable for those scenes. Dynamic selection of the content can be used to apply this method whenever the content is suitable and to switch to another method for the rest of the content, e.g. fast moving scenes. The difference between the two modes is a binary decision and is difficult to reproduce perfectly in the same manner during extraction as has been used during embedding if the content has undergone transformations such as blurring or frame reduction. If uncertain, the extraction process can assume that the pattern has been embedded in a single static phase without variation, because the multiphase mark can typically be read at least in part with a regular detector albeit with reduced strength. A dynamic approach can be particularly effective when used in combination with a watermark extraction process that relies on combination of frames to accentuate the embedded watermark. For example, the method described in U.S. Patent 7,430,302 entitled "Covert and Robust Mark for Media Identification" to Thorwirth can be used to embed watermarks in portions of the content that exceed a predetermined amount of variation, and a multiphase watermark used with the remaining content. During extraction, the similarity of the content can be inspected to identify content likely to contain a multiphase watermark and extraction processes used to create difference frames. The complete watermark could then be obtained by combining the difference frames with the remaining frames of the video sequence in the manner described in U.S. Patent 7,430,302, the disclosure of which is incorporated herein by reference in its entirety. In many embodiments, use of a human recognizable watermark pattern as the basis of the multiphase watermark can provide similar benefits to those outlined in U.S. Patent 7,430,302 irrespective of whether the multiphase watermark is used in combination with the watermarking process described in U.S. Patent 7,430,302.

Improving the Selection Process

[0097] In order to retrieve the complete watermark pattern, some knowledge about the phase is required. If the start of a cycle and the length of the phases are known, the distribution of the watermark pattern in different phases is known and can be used during extraction. Referring to FIG. 3 as an example, if it is known that a frame contains the part of the watermarking pattern represented with the letters ABC and the difference is made between that frame and a frame one cycle length (i.e. three frames) later, then we know that the selected frames contains the watermark sub pattern DEF and the difference is ABC-DEF where the second sub pattern DEF is a negative representation and can be inverted to obtain the positive representation. This however is practically not always possible, as without the presence of the original media object and embedding parameters, the start location of the phase is not known. In many instances, the start of the video might have been changed during processing before the video is analyzed to read the watermark pattern.

[0098] In several embodiments, no assumptions are made concerning the starting point of the video or the phase length. Even without knowledge of the start position or the geometric location of each sub pattern, the length of the phase is still required in order to be able to select frames that contain different phases as the length of the phase is assumed to be known. This is generally the case, in particular if the phases are divided in time, since the time domain is typically maintained to preserve reasonable quality of the media object. However there are processing errors or modifications that can occur and hinder the identification of phase distances such as re-sampling of the content results in a different sample rate, merged samples, omitted samples, cropped samples and added samples (for digital video: frame drops, frame addition, frame rate change, removing frames and interlacing). These modifications can reduce the quality of the extraction. One embodiment of this invention aims to derive a fixed pattern from a media object using a sequence of changing patterns. If the combination of the phases is not well reproduced and differs between embedding and extraction, the extraction is not synchronized and the approach results in an extracted watermark pattern that does not have an optimal strength. Several possible solutions to prevent or alleviate the problem are described below. [0099] A change in the sample rate that does not change the content duration can be understood and accounted for, if the embedding phase is independent of the sampling rate, e.g. the length of the phase for video is a fixed number of milliseconds rather than a fixed number of frames. The process of translating from milliseconds to the length in frames is performed during embedding and also during extraction. It is to be noted that the frame rate, in frames per second is provided with the file in which the frame rate was changed as a requirement to play it back correctly.

[00100] To identify the start of the phase, a registration of a fixed point in time is required. The start point of the phase is useful when absolute values of differences between sub patterns are used. The start of a movie is typically not practical, as a copy of that content might start at a different time. A useful temporal position that is relatively robust against distortions introduced to the video is a detection of a temporal discontinuity. At a detection of that point in time, a new phase is started and the embedder and the detector are using a same or similar algorithm to locate that point in time. This is useful to establish synchronization in time without requiring knowledge about the unmarked video for identifying these points in time. To identify the temporal discontinuity a peak in the change in global properties of the media object over time is useful. For video this can be a change of average luminance over time. The value can be compared to a pre determined threshold in order to make a decision if a temporal discontinuity has been detected. Other well known technologies used for scene change detection also apply as a robust differentiator for temporal locations.

[00101] In order to maintain coherent watermark extraction, intermediate extraction results can be compared for the continuity of the watermark pattern. The media object is therefore divided into several chunks. The extraction is performed on each of the chunks using the extraction process described above and the samples of the phases are combined with the assumed period of a sample. For each chunk, all possible combinations are evaluated and the result is several extraction results for each chunk. The results are then compared for similarity and only similar results are further combined. This approach is in particular useful in the case where single phases are extracted only, as explained in the rolling phase extraction. In this case, the number of extraction results is equivalent to the number of phases. The comparison can be performed by a human operator or by automated comparisons such as the well known correlation approach. The decision of matching extraction results can be biased by the assumption that a change in the time domain occurs infrequently and that it is likely that the shift in the phase will be small. The chunks might be as small as a single phase for content that is not degraded other than in the time domain. In one embodiment, each chunk corresponds to three seconds of content and is, therefore, likely to contain a meaningful watermark pattern even if the content is degraded. The comparison also does not necessarily have to be performed between two equally sized chunks. The size of the chunks may vary, depending on the properties of the content and the comparison result may be performed between the latest extracted chunk and the combined extraction result of all previous chunks.

[00102] Using extraction with no absolute value computation may be susceptible to alterations that change the content duration and its time domain sequence. The alterations occur during frame rate changes that do not maintain the content duration, during frame drops and cut outs of sections or scenes. Correlation techniques applied to the extraction results for individual chunks of the content can be used to automatically account for and eliminate the negative effect of frame drops and frame cut outs. A frame rate change causes phase shifts equally spaced in time which are particularly impairing to the watermark extraction with no absolute value computation. This impediment can be corrected for by introducing a phase shift at time intervals corresponding to the particular frame rate change into the extraction process. Unidentifiable frame rate changes that do not maintain the content duration may be corrected with the aid of correlation applied to the extraction results with correction for various assumed common frame rate conversions. The extraction results of chunks of media are compared. The extraction results will exhibit greater similarity when the extracted pattern is stronger. This is the case if the assumed frame rate conversion is correct and the phase is estimated correctly. Thus, comparative analysis of the correlation values for different phases and for different chunks of the content can be used to characterize frame rate convergence and attain optimal extraction results.

[00103] All correlation results can be improved by limiting the correlation to the geometric portion of frames containing the watermark pattern only, which reduces the amount of background noise outside the watermark pattern that does not contribute positively to the comparison and correlation. Another method for improving correlation for watermark extraction results with no absolute value computation can be the use of extraction results independently derived with other extraction methods such as extraction with absolute value computation. These different extraction results can be compared, combined or used as a seed to the correlation process of extraction results for following chunks of the content. Finally other marking technologies might be deployed that embed a reference signal in time, allowing the detection of the length of the cycle, phases and the start points.

Extension of the Multiphase Watermark Approach

[00104] The concept of multiphase watermark has been described above using the concept of marking frames with occasional references to other audio and other representations of video. The extension of multiphase watermarking to other media objects and to additional embedding domains is described below. [00105] A fundamental assumption of the multiphase approach is that related media units are similar and that a watermark pattern that is divided into sub pattern which are distributed over several phases, introduce additional differences between media units that can later be emphasized by using this assumed similarity of the media units. In other words, the un- watermarked value of a media unit is estimated by observation of a related media unit. The difference between the two averaged with the differences between other marked media units reveals the watermark.

[00106] The media units, the way they are encoded and the relationship between the media units that is used in the recovery of the sub-patterns can vary, enabling many ways to embed a multiphase watermark. Several examples for encoded media units and their relationship are listed below with a description of how to use them for embedding information in a multiphase approach.

[00107] Frames in video sequences are represented as a sequence of still images and can be encoded in a baseband domain and every picture element or pixel represents a point of the image and is encoded as a value representing its color and or luminance content. Frame as media units have been used in most of the examples above where their relationship in time and their pixel representation has been used to explain various multiphase concepts that have been applied to those media units. Instead of video frames, individual pixels or groups of pixels can be used as media units and their relationship in space can be used as the bass of a multiphase watermarking approach. The variation between neighboring pixels is typically small, in particular in high resolution video content. Information that is added to some pixels can be estimated by subtracting neighboring pixels. The watermark pattern in this case is a sequence of N bits, that is distributed over P*N pixels, with P being the number of phases in a cycle. In one embodiment, the P*N pixels will be distributed in groups of P such that pixels of different phases will be located in close proximity. This approach of using individual pixels to store information resembles the prior art technique known as least significant bit approach but exhibits slightly better robustness while also allowing for a large payload. When pixels are grouped the robustness is increased while the payload capacity is reduced. [00108] The corresponding approach to use differences between frame pixels is the application of a P*N bit pattern to PCM samples of audio where the phases are neighboring samples of digitized audio.

[00109] The encoding of media units may not be in individual pixels or PCM samples but the same information may be represented as frequencies. Common frequency transformations include DCT, FFT and Wavelet and can be applied to audio and video. For video frequencies used in a multiphase approach, the assumption can be used that the frequency distribution between units in time (nearby video frames) does not vary and can serve as estimation. Here, the frequency components of frames are compared to identify the variations in them which emphasize a varying mark embedded in multiple phases.

[00110] In the same way, audio frequency information can be used when audio is grouped into intervals that are analyzed for frequency content and compared to neighboring intervals in time in order to understand differences, which may be part of a watermark pattern. The continuous nature of audio makes it more likely that the interval boundaries are not maintained during operations such as editing as a natural interval like video frames does not exist. In these instances, employing a rolling phase watermark and correlation can be particularly useful in the recovery of the watermark.

[00111] The image frequencies may also be used in a block based manner, exploiting similarities between neighboring image blocks in the frequency domain. Images and video frames are often stored in image blocks stored as frequencies, e.g. in formats such as jpg and mpeg and can easily be modified in this format.

[00112] A media unit in which a multiphase mark is embedded need not be part of the content, in some embodiments it is derived information such as the amplitude or intensity that is derived from the sound volume or average sound pressure levels. In a number of embodiments, the audio track is modified to embed a multiphase watermark that is extracted from variations of the sound volume.

[00113] Another derived media unit is the intensity of a video sequence over time, measured as the average luminance of a frame or of a group of frames. In order to improve a watermarking scheme that embeds a watermark pattern as variations in average luminance or flicker in the video, the modification of the luminance associated with a multiphase watermark only occurs in some frame intervals or some areas of the frames, while other frame intervals or areas are not modified in order to serve as an estimator for extraction to emphasize the embedded signal in contrast to the underlying content. The extraction process in this embodiment uses the average luminance of the unmarked frames and subtracts the average luminance of the marked frames. As the average luminance does not typically vary 'significantly in a video sequence, the remaining differences are typically small and largely composed of the watermark pattern. [00114] The variations above show how the multiphase approach can be used in different embedding domains. An actual implementation of a multiphase watermark can use any of these domains or any other domain in which a watermark can be embedded. The addition of a specific frequency watermark pattern to audio and video for example can be performed in the special domain for video and temporal domain for audio by adding frequencies to the media. This may be helpful to embed a watermark pattern in the desired embedding domain without processing intensive transformation in that domain.

Applications for Multiphase Watermarks

[00115] The systems and techniques described above can be used in a virtually limitless range of applications. The following are provided as a series of examples of applications in which systems and processes that utilize multiphase watermarks in accordance with embodiments of the invention can be useful. Usefulness is established when a multiphase adaptation is performed using an existing watermarking approach in order to improve the strength and security of the embedded watermark pattern, i.e. to read the mark even after severe degradation of the content or to make it more difficult for an attacker with the intent to remove the mark to do so successfully.

Tracking Unauthorized Distribution of Secret or Copyrighted Information [00116] A problem that faces many industries is the unauthorized distribution of information. Systems and processes in accordance with embodiments of the present invention can be used to embed multiphase watermarks in media objects at the time of reception or display of the media. Each distributed copy can be uniquely marked with information such as a recipient identification number and a time stamp and, if the copy is publicly available or in the possession of an entity or individual that is not authorized to possess the information, the information can be uncovered and the entity or person that is the recipient of the media object and the likely source of the unauthorized distribution can be identified.

[00117] In many instances, the secret or copyrighted information is passed between several different entities and/or individuals during production and authorized distribution. In several embodiments, the point from which the information was distributed without authorization can be ascertained by embedding a mark associated with the last recipient of the information prior to delivery or display. The entity or individual that is responsible for the unauthorized distribution can then be identified based upon the last watermark pattern added to the media. [00118] A common instance, in which copyrighted information is communicated, is the distribution of copyrighted media objects via a network to a media player. In many embodiments, the player is a consumer electronics device such as a set top box or a personal computer. The copyrighted media object is typically distributed to the player in a compressed and encrypted form. A multiphase watermark can be embedded in a media object in accordance with embodiments of the invention. The multiphase watermark can contain information relating to the owner of the player and information identifying the time of transmission or playback. If the recipient of the information is known, the information to be embedded can be generated by the server (or head end) providing the media. The embedded information can also be stored by the server in a database that contains additional information about the transaction, such as the user's billing information and details about the receiving device. In other embodiments, the player maintains information such as player identification number and time, which is embedded as a multiphase watermark during storage and/or playback.

[00119] Another instance in which unauthorized distribution is a common problem is in the production of media. During production, content is particularly vulnerable to unauthorized distribution that can cause considerable damage to the producer of the media. In many embodiments, multiphase watermarks are embedded in a media object during various stages of production that identify the recipient of the media object and the time of the receipt of the media object. If a copy is made publicly available, the multiphase watermark can be extracted and the responsible person or entity identified.

[00120] In many of the embodiments outlined above, a perceptual model can be used to hide the pattern in addition to the creation of different phases. A perceptual model can be created for the distributed media object and stored or distributed with the media object. The perceptual model can then be used as marks are embedded in the media. In other embodiments, a perceptual model is created every time a mark is embedded in the media.

Proof of Ownership

[00121] Once a media object is subject to public distribution, proving ownership of copyright in the media object can be problematic. In many embodiments, multiphase watermarks are embedded in the media object to prove ownership of the content. In a number of embodiments, the mark includes ownership information or identification for copyrighted content. This multiphase watermark can be automatically read and its presence or absence can be evaluated to restrict or allow distribution or to verify that a source distributing the media is a legitimate distributor of the media. It can also be used, so that entities or individuals interested in obtaining, rights with respect to the content can identify the owner of the content by extracting the multiphase watermark. Another use for a multiphase watermark identifying the owner is to automatically identify content in a publicly available database, e.g., accessible through the Internet for content owned by a specific owner or group of owners to prevent further distribution. The multiphase watermark can also be used to prove ownership in case of a dispute.

Storing Robust Meta-Information

[00122] During long term storage and archival, information stored with media objects can be lost either because it is deleted accidentally or because it cannot be read. Many embodiments of the invention are configured to store information concerning media using marks. Storing information using marks can enable the information to be retrieved after format change and does not require additional storage space.

Copy Control

[00123] In many embodiments, a multiphase watermark mark that is machine readable can be embedded into a media object. The multiphase watermark can then be used by a player to control playback and/or recording of the media object. The player can uncover the multiphase watermark and ascertain the authorizations required to play the media. If the player has the appropriate authorizations, then the player can play the media object or refuse playback otherwise. Broadcast Monitoring

[00124] Machine readable multiphase watermarks in accordance with embodiments of the present invention can also be embedded in media objects broadcast by a television or radio station. The machine readable multiphase watermarks can then be used by receiver devices to automatically record the content that was broadcast and to track the frequency and time of broadcast. The multiphase watermark embedded in the media object can be distinct for each work and broadcast.

Secret Communication

[00125] Multiphase watermarks can be used to transmit secret information using media objects. While the transmission of the media object can be observed, the fact that the media object is marked is not obvious and can be used to transmit information without allowing others to observe that information other than the media content is being transmitted.

Identification of Publicly Displayed Media Objects

[00126] Media objects that are publicly displayed are frequently the subject of rerecording. In many embodiments, a multiphase watermark is embedded in a publicly displayed media object that identifies the time and/or place of public display. Should the media object be rerecorded during the public display, then the multiphase watermark is embedded in the unauthorized rerecording and uncovering this information can be helpful in preventing future rerecording of publicly displayed media at the venue in which the rerecording was made.

Content Authentication

[00127] Digital media is not only used for entertainment purposes but also recorded for the purpose of documentation. This is an example where authentication of such media objects increases it value as evidence and credibility that it has not been modified. This is particularly important for digital media, which can be manipulated more easily than analogue content, since tools are readily available and reproduction is flawless. Multiphase watermarks in accordance with embodiments of the invention can be used to embed a fragile message that is affected by many manipulations and if the multiphase watermark is not perfectly restored it can be concluded that the media has been manipulated. [00128] While the above description contains many specific embodiments of the invention, these should not be construed as limitations on the scope of the invention, but rather as an example of one embodiment thereof. For example, while the phases have frequently been described as frames of a digital video, the application is not limited to the phase as a number of frames. Multiphase watermarks can also be applied when phases are blocks of an image, such as 8 x 8 pixel blocks, fields of a frame, audio samples in an audio piece, frequencies of an image, or audio piece or frequencies in time of a digital video. Accordingly the watermark pattern is not required to be an image pattern applied to frames of a video, as is used as example in the explanation of various features. The watermark pattern can be any pattern suitable for combination with a media object such as a set of frequencies applied to video frames or a set of audio frequencies that create a pattern in their intensity or relation to each other. In addition, almost all existing watermarking approaches can be improved by taking a multiphase approach in accordance with an embodiment of the invention. Most approaches can be abstracted as embedding a watermark pattern, therefore, multiphase watermarking techniques can be applied to the embedding and extraction processes. The pattern would be created differently than the patterns described above e.g. in a frequency domain or through modulation of global parameters over time. However, the approaches described above are equally applicable to the extraction or interpretation of such marks by improving the signal to noise ratio between the sub patterns and the underlying content. Accordingly, the scope of the invention should be determined not by the embodiments illustrated, but by the appended claims and their equivalents.

Claims

What is claimed is:

1. A method of imperceptibly and robustly embedding a watermark pattern in a distributed fashion within a media object, comprising: dividing a watermark pattern into a plurality of sub patterns wherein at least one of the sub patterns is missing a portion of the complete watermark pattern; and combining each sub pattern with a different media unit of the media object in a repeated embedding cycle, thereby creating a modified media object with modified media units; wherein the complete watermark pattern can be recovered from the modified media object by observing differences between modified media units in which different sub patterns are embedded.

2. The method of claim 1 , wherein the media object is digital audio and the media units are audio intervals.

3. The method of claim 1, wherein the media object is a video sequence and the media units are video intervals.

4. The method of claim 3, wherein the video interval corresponds to a single frame of video.

5. The method of claim 4, wherein the watermark pattern includes human recognizable symbols.

6. A method of claim 4, wherein: the watermark pattern is divided into a plurality of sub patterns, where each sub pattern resembles film grain and at least a portion of the watermark pattern is represented by differences in the film grain between different sub patterns; and the sub patterns are combined within the frames of the video sequence so as to appear as film grain to a human observer.

7. The method of claim 1 , wherein combining each sub pattern with a different media unit further comprises modifying the sub pattern according to the content of the media unit using a perceptual model before combination.

8. The method of claim 3, wherein: combining each sub pattern with a different media unit of the media object in a repeated embedding cycle, thereby creating a modified media object with modified media units further comprises: determining whether the content in a sequence of consecutive media units contains a level of variation below a predetermined threshold; when the level of variation is below the predetermined threshold, combining each sub pattern with a different media unit in the sequence of consecutive media units in a repeated embedding cycle; and when the level of variation is above the predetermined threshold, embedding the watermark pattern in the sequence of consecutive media units using an alternative embedding process.

9. The method of claim 1, wherein the watermark pattern is represented in a frequency domain.

10. The method of claim 1 , wherein creating a plurality of sub patterns using the watermark pattern further comprises creating at least one complimentary pair of sub patterns, wherein each sub pattern in the complimentary pair includes a different portion of the watermark pattern and the pair can be combined to recreate the complete watermark pattern.

1 1. The method of claim 1 , wherein the embedding cycle involves embedding different sub-patterns in media units separated by a predetermined temporal distance.

12. The method of claim 1 1 , wherein the predetermined temporal distance varies in a predetermined way as the embedding cycle repeats.

13. The method of claim 11 , wherein: the media object is a video sequence and the media units are individual frames of video; and the predetermined temporal distance between media units is chosen to minimize flicker in the modified video sequence.

14. The method of claim 1, wherein the embedding cycle is a rolling cycle wherein: creating a plurality of sub patterns using the watermark pattern further comprises: creating a first sub pattern using a first portion of the watermark pattern; creating additional sub patterns by combining an additional portion of the watermark pattern with the previous combination until a sub pattern is created that includes the entire watermark pattern; creating additional sub patterns by removing a portion of the watermark pattern from the combination in the order the portions of the watermark pattern were originally added to the combination until a sub pattern is obtained that does not contain any portion of the watermark pattern; and wherein the sub patterns form complimentary pairs, where each complimentary pair includes a different portion of the watermark pattern, and the pair can be combined to recreate the complete watermark pattern; and combining each sub pattern with a different media unit in a repeated embedding cycle, further comprises: combining each of the sequence of sub patterns with each media unit in an order corresponding to the order in which portions of the watermark pattern were added and removed to create the sub patterns.

15. The method of claim 1, wherein the separation of the watermark pattern into sub patterns varies between embedding cycles and the sub patterns can be chosen on pseudorandom basis.

16. A method of imperceptibly and robustly embedding a watermark pattern in a distributed fashion within a video sequence, comprising: dividing a watermark pattern into a positive pattern including the complete watermark and a negative pattern including an inverse of the complete watermark, such that the positive and negative patterns cancel when combined; and combining the positive and negative patterns with different frames of the video sequence in a repeated embedding cycle, thereby creating a modified video sequence with the modified frames; wherein the complete watermark pattern can be recovered from the modified video sequence by observing differences between a modified frame in which a positive pattern is embedded and a modified frame in which the negative pattern is embedded.

17. A method of emphasizing a watermark pattern embedded in a distributed fashion within a marked media object, where the marked media object is generated by combining sub patterns of the watermark pattern with media units of the media object in a repeated embedding cycle, comprising: selecting at least one pair of media units from the media object, where each of the pair is likely to include a different sub pattern; subtracting one media unit in the pair from the other media unit to create a difference media unit in which the sub patterns are emphasized relative to the content of the media units; and estimating the watermark pattern using at least the difference media unit.

18. The method of claim 17, further comprising: selecting multiple pairs of media units, where the media units in each pair are likely to include different sub patterns; subtracting one media unit from the other media unit within each pair of media units to create difference media units in which the sub patterns are emphasized relative to the content of the media units; combining the difference media units to create an accumulated media unit in which the parts of the watermark pattern from each of the sub patterns is emphasized; and estimating the watermark pattern using the accumulated media unit.

19. The method of claim 18, wherein the absolute differences of the media units are combined to create the accumulated media unit.

20. The method of claim 18, wherein selecting at least one pair of media units in the video sequence that is likely to include different sub patterns further comprises selecting pairs of media units that are likely to include complimentary sub patterns.

21. The method of claim 18, wherein the media object is digital audio and the media units are intervals of audio.

22. The method of claim 18, wherein the watermark pattern is represented in a frequency domain

23. The method of claim 18, wherein the media object is a video sequence and the media units are individual frames of video.

24. The method of claim 23, wherein the watermark pattern is a human recognizable image.

25. The method of claim 18, wherein the marked media object contains consecutive media units in which the watermark pattern is embedded using an embedding process that does not utilize sub patterns and the method of recovering the watermark pattern further comprises: determining whether the content in a sequence of consecutive media units contains a level of variation below a predetermined threshold; when the level of variation is below the predetermined threshold, recovering information concerning the watermark pattern by: selecting multiple pairs of media units from the sequence of consecutive media units, where the media units in each pair are likely to include different sub patterns; subtracting one media unit from the other media unit within each pair of media units to create difference media units in which the sub patterns are emphasized relative to the content of the media units; combining the difference media units to create an accumulated media unit in which the parts of the watermark pattern from each of the sub patterns is emphasized; and estimating the watermark pattern using the accumulated media unit; when the level of variation is above the predetermined threshold, recovering information concerning the watermark pattern from the sequence of consecutive media units using an alternative extraction process; and combining the recovered information to estimate the watermark pattern.

26. The method of claim 25, wherein: the media object is a video sequence, the media units are individual frames of video, and the watermark pattern is a human recognizable image; and the alternative extraction process comprises combining frames of video in a predetermined manner that results in an accumulated image in which the parts of the watermark pattern present in the frames of video are emphasized.

27. The method of claim 18, wherein: selecting at least one pair of media units comprises: selecting pairs of media units separated by an initial temporal distance and generating an initial accumulated media unit over a first portion of the media object; generating multiple accumulated media units by selecting pairs of media units from a second portion of the media object, wherein each of the multiple accumulated media units is generated by selecting a different temporal distances between the pairs of media units selected from the second portion of the media object; and comparing each of the multiple accumulated media units with the initial accumulated media unit to select the temporal distance that yields the greatest continuity of results over the second portion of the media object; and combining the difference media units to create an accumulated media unit in which the parts of the watermark pattern from each of the sub patterns is emphasized comprises: combining the initial accumulated media unit with the accumulated media unit generated with the temporal distance that yields the greatest continuity to create an accumulated media unit in which the parts of the watermark pattern from each of the sub patterns is emphasized.

28. The method of claim 18, wherein: selecting at least one pair of media units comprises: selecting pairs of media units separated by a selected temporal distance and generating an initial accumulated media unit over a first portion of the media object; generating multiple accumulated media units by selecting pairs of media units separated by the selected temporal distance from within a second portion of the media object, where each of the multiple accumulated media units is generated by selecting a different start time from which to commence selecting the pairs of media units from within the second portion of the media object; and comparing each of the multiple accumulated media units with the initial accumulated media unit to select the start time that yields the greatest continuity of results over the second portion of the media object; and combining the difference media units to create an accumulated media unit in which the parts of the watermark pattern from each of the sub patterns is emphasized comprises: combining the initial accumulated media unit with the accumulated media unit generated with the start time that yields the greatest continuity to create an accumulated media unit in which the parts of the watermark pattern from each of the sub patterns is emphasized.

29. The method of claim 28, wherein the media object is a video sequence and the comparison is performed with respect to results accumulated over portions of the video sequence corresponding to three seconds of video.

30. The method of claim 17, wherein selecting at least one pair of media units from the media object further comprises selecting a pair of media units separated by a predetermined temporal distance.