KR20080043323A - Coefficient modification for video watermarking - Google Patents

Abstract

A method and system for watermarking video images including generating a payload, selecting sets of coefficients, modifying coefficients and embedding said watermark by enforcing relationships between property values of selected sets of coefficients with a volume of video are described. The modified coefficients replace the selected sets of coefficients.

Description

Change coefficients for video watermarking {COEFFICIENT MODIFICATION FOR VIDEO WATERMARKING}

The present invention relates to the watermarking of video content, and in particular to embedding and detecting watermarks in digital cinema applications.

The video includes a spatial axis and a time axis. Images (and likewise video frames) can appear in the spatial domain or in the transform domain. In the spatial domain, also called the 'baseband' domain, the image appears as a pixel value grid. The transform domain representation of the pixelated (ie discrete) image can be calculated from the mathematical transform of the spatial domain image. In general, this conversion can be completely reversible or at least without the significant loss of information. There are several transform regions, the best known of which are FFT (fast Fourier transform), DCT (discrete cosine transform) used in JPEG compression algorithm, and DWT (discrete wavelet transform) used in JPEG2000 compression algorithm. One advantage of presenting content in the transform domain is that such representation may generally be more compact than baseband representation for similar conceptual qualities. Watermarking methods exist for embedding watermarks in the transform region as well as in the baseband.

Video or video images are suitable for various watermarking approaches. This approach to video watermarking can be grouped into three categories based on whether this approach selects a spatial structure, a temporal structure, or a global three-dimensional structure of video for watermarking.

The spatial video watermarking algorithm extends still image watermarking to video watermarking by inserting marks per frame using the existing image watermarking algorithm. In the prior art, per-frame watermarks are repeated in each frame over a certain interval, which interval is arbitrary and can be from a few frames to the entire video. On the detector side, it is advantageous for the power signal-to-noise ratio (PSNR) to have the same watermark pattern repeated on multiple successive frames. However, if all frames have the same watermark pattern, special care may need to be taken to avoid vulnerabilities to possible frame collusion attacks. On the other hand, if the watermark changes for every frame, it may be more difficult to detect, while causing flicker defects and still being vulnerable to collusion attack in stable areas of the video.

As an improvement, it is not necessary to watermark every frame. In the prior art, only automatically selected 'key frames' (and some frames around the key frames) are watermarked. The key frame is a stable frame found between two boundary shot frames and can be reliably positioned again even after a frame rate change. Watermarking only key frames not only reduces the pressure on fidelity constraints, but can also result in better security and less computational strength.

Spatial area watermarks can benefit from still image watermarking techniques that are resistant to geometric transformations, for example by using geometric invariant watermarks, or by duplicating a watermark of a tiled pattern or by using a template of Fourier region, It is difficult to reverse due to geometrical transformations and screen curvatures that occur during camcorder capture of a casted movie. Furthermore, these two approaches are not safe for signal processing attacks, for example, the template of the Fourier region can be easily removed. Thus, the spatial area watermark can be detected more easily and securely if the original content is used for registration. In the prior art, a semi-automatic registration method is used, which matches feature points in the original frame with feature points in the extracted frame. For projection on a flat screen, at least four reference points must be matched to reverse the transformation. The operator manually selects at least four feature points from a set of pre-computed feature points. Two-level registration can be done completely automatically: first in the time domain, then in the spatial domain. A database for frame signatures (also called fingerprints, soft hashes or message digests) is accessed by the watermark detector to match the extracted key frame with the corresponding original frame. The latter is then used for automatic space registration of test frames.

However, it should be noted that the calculation for the selection of the key frame requires an upcoming frame, which is not available at the time of watermark embedding for the real time application. An alternative method is to maintain a constant time delay between frame processing and playback.

Prior art time watermarking schemes only use the time base to insert a watermark by changing the global luminance within each frame. This not only makes the watermark inherently strong in geometric distortion, but also simplifies the watermark reading after camcorder attack. The power of the watermark for temporal lowpass filtering (usually applied when unflashing the camcorder's video) can be improved using other methods known in the art. However, watermarks can be weak for temporal de-synchronization (especially after frame editing). However, synchronization can also be restored by matching key frames between the desynchronized video and the original video.

Two previous approaches (spatial or temporal watermarking) use one or two of the three available dimensions for watermarking. The absence of one or two watermark structures in the three available dimensions in the video results in the suboptimal use of space available for the watermark. The method described in Bloom et al. US Pat. No. 6,885,757 " Method and Apparatus for Providing an Asymmetric Watermark Carrier " fully utilizes the structure of the video. In the spread-spectrum method of the above patent, this technique is obviously powerful and safe, but the detector must synchronize the test video with the original video before detection.

Aspects of the present invention involve pseudo-randomly inserting constraint-based relationships between attribute values of constant coefficients over consecutive frames or within a single frame. This relationship encodes watermark information.

'Coefficient' is represented as a set of data elements, which includes video, image, or audio data. The term 'content' will be used as a generic term for indicating any set of data elements. If the content is in the baseband region, the coefficient will be expressed as a 'baseband coefficient'. If the content is in the transform region, the coefficient will be indicated as 'transform coefficient'. For example, if each frame of an image, or video, appears within a spatial region, the pixel is an image coefficient. If an image frame appears within the transform region, the value of the transformed image is the image coefficient.

The present invention addresses DWT for JPEG200, particularly in digital cinema applications. The DWT of the pixelated image is calculated by the continuous use of the vertical and horizontal low-pass and high-pass filters into the image pixels, the resulting value being called the 'wavelet coefficient'. A wavelet is an oscillating waveform that lasts for only one or several cycles. At each iteration, the low-pass filtered wavelet coefficients of the previous iteration are decimated, followed by a low-pass vertical filter and a high-pass vertical filter, and the result of this process is low-pass horizontal and high-pass. Passing is passed through a horizontal filter. The resulting set of coefficients is grouped into four 'subbands', LL, LH, HL and HH subbands.

That is, the LL, LH, HL, and HH coefficients are low-pass vertical / low pass horizontal filters, low-pass-vertical / high-pass horizontal filters, high-pass vertical / low-pass horizontal filters, high-pass vertical / The coefficient resulting from the continuous use of the high-pass horizontal filter into the image.

An image may have multiple channels (or components) corresponding to different native colors. If the image is grayscale, this image has only one channel representing the luminance component. In general, the image is colored, in which case three channels (sometimes different numbers of channels are used) are typically used to represent different color components. Each of the three channels can represent red, green, and blue components, in which case the image appears in the RGB color space, but many other color spaces can be used. If the image has multiple channels, the DWT is typically calculated separately on each color channel.

Each iteration corresponds to a constant 'layer' or 'level' of the coefficients. The first layer of coefficients corresponds to the highest resolution level of the image, while the final layer corresponds to the lowest resolution level. 1 is a video representation within one component of a five-level wavelet transform. Units 105-120 are video frames. Unit 125 points to the LL subband coefficient at the lowest resolution. Unit 125a is a frame (f = 0), channel (c = 0), subband (b = 0), resolution level (l = 0), and position (x and y = 0), where (f, c, coefficients in l, b, x, y).

In order to best use the 3D structure of video, the present invention utilizes a time axis and a space axis. Since spatial registration is difficult to achieve for a movie after projecting and capturing, the present invention utilizes a very low spatial frequency or a global characteristic of low spatial frequency, which is less sensitive to geometric distortions for spatial registration. The time frequency is more easily recovered because most of the transformations occurring during the attack are time-linear.

In the present invention, the low-resolution wavelet coefficients of the video are directly watermarked. Since the number of pixels in a frame is about 1000 times larger than the number of lowest resolution wavelet coefficients, the number of operations is potentially much smaller in the present invention.

A method and system for watermarking a video image is described, wherein the video image is generated by generating a watermark, and by performing a relationship between the generated watermark and the characteristic value of the selected coefficient set. It includes inserting in. The watermark is thus adaptively inserted in the video volume. A method and system for watermarking a video image is also described, which includes selecting a coefficient set and performing an association between the video volume and characteristic values of the selected coefficient set. A method and system for watermarking a video image is also described, which includes generating a payload, selecting a coefficient set, changing a coefficient, and an association between video volume and characteristic values of the selected coefficient set. And embedding the watermark. The changed coefficient replaces the selected coefficient set.

A method and system for detecting a watermark in a video image are described, which includes preparing a signal, extracting and calculating characteristic values, detecting bit values, and decoding a payload, wherein The payload is a string of bits created and inserted by performing an association between feature values within the video volume. A method and system for detecting a watermark within a video image is also described, which includes preparing a signal and decoding the payload, where the payload is implemented by performing an association between characteristic values within the video volume. Generated and inserted bit string. A method and system for detecting a watermark in a video volume is also described, which includes preparing a signal, extracting and calculating characteristic values, and detecting bit values.

The invention may be implemented in hardware, firmware, FPGAs, ASICs, etc., but is best implemented in software residing within a computer or processing device, which device may be a server, a mobile device or any equivalent thereof. The method is best implemented / implemented by programming the steps and storing the program on a computer readable medium. If the speed required for real-time processing requires hardware for one of the more sequences of steps, the hardware solution for all or any part of the processes and steps described herein is easily implemented without any loss of generality. Can be. This hardware solution can then be inserted into a computer or processing device, such as but not limited to a server or mobile device. In an implementation for real-time watermarking JPEG2000 images for digital cinema applications, the JPEG2000 decoder in the digital cinema server or projector passes the coefficients of the lowest resolution level of each frame to the watermarking insertion module. This insertion module changes the received coefficients and returns them to the decoder for further decoding. This transfer, watermarking and return of the coefficients is performed in real time.

The invention is best understood from the following detailed description when read in conjunction with the accompanying drawings. The figure includes the figures briefly described below, wherein like-numbers on the figures indicate similar elements.

1 illustrates a video representation within one component of a five-level wavelet transform.

2 is a flowchart illustrating a payload generation step of watermarking.

3 is a flowchart showing a coefficient selection step of watermarking.

4 is a flowchart showing a coefficient changing step of watermarking.

FIG. 5 shows a video frame reconstructed from video frames at full resolution and coefficients at resolution level 5. FIG.

6 is a block diagram of watermarking in a D-cinema server (media block).

7 is a flowchart illustrating video watermark detection.

8 is a flowchart showing signal preparation for video watermark detection.

9 illustrates a cross-correlation function.

Fig. 10 is a flowchart showing detection of a bit value at the time of video watermark detection.

11 illustrates an accumulated signal.

Many applications require real-time watermark embedding, such as session-based watermark embedding for set-top boxes and for digital cinema servers (also called media blocks) or projectors. While quite obvious, it is worth mentioning that at some time, it makes it difficult to apply a watermarking method that uses frames later in time. Offline pre-computation (eg for the location or strength of the watermark) should preferably be avoided. There are several reasons for this, but two most important reasons are: potential security leaks (currently generated watermarking algorithms are generally less secure if the attacker knows the full details of the insertion algorithm), and impractical to be.

In most applications, digitally watermarked content units generally undergo some change between the time of insertion and the time of detection. This change is termed 'attack' because this change generally degrades the watermark and makes the detection of the watermark more difficult. If such an attack is expected to occur naturally during application, the attack is considered 'unintentional'. Examples of unintentional attacks include: (1) watermarked images that are cropped, scaled, JPEG compressed, filtered, etc. (2) converted to NTSC / PAL SECAM for viewing on television displays, MPEG or DIVX may be watermarked video that is compressed, resampled, or the like. On the other hand, if the attack is deliberately done to remove the watermark or spoil the detection of the watermark (ie, the watermark is still in the content but cannot be retrieved by the detector), the attack is 'intentional' and the attack The party performing this action is the 'copyright infringer'. Intentional attacks generally aim to maximize the likelihood of making the watermark unreadable, while minimizing conceptual damage to the content, an example of an attack being applied to content to make it very difficult to synchronize with the detector of the content. It may be a small, unrecognizable combination of local rotation / scaling and / or line removal / addition (most watermark detectors are sensitive to de-synchronization). A tool is present on the Internet, such as Stirmark (http://www.petitcolas.net/fabien/watermarking/stirmark) for the above attack purposes.

In the case of a so-called 'camcorder attack', this is carried out by a person who illegally captures a movie during playback in a theater, which is considered unintentional even if the party is performing an illegal act. In fact, movie capture is not done for the purpose of removing the watermark. However, after capturing the movie, the person can run an additional process on the captured video to ensure that the watermark can no longer be detected in the content.

For example, session-based watermarks for digital cinema must withstand the following attacks: resizing, letterboxing, aperture control, low-pass filtering and anti-aliasing, brick wall filtering, digital video Add noise reduction filtering, frame-swapping, compression, scaling, cropping, overwriting, noise, and other conversions.

Camcorder attack includes the following attacks in a sequential order: camcorder capture, de-interlacing, cropping, blinking and compression. In particular, camcorder capture causes significant spatial distortion. The present invention focuses on a camcorder attack, which is because the watermarks that endure the camcorder attack will withstand most other unintentional attacks such as screener copy, television, etc. Because it is generally recognized. However, it is also important that the watermark withstands other attacks. Frames of video are generally interlaced for playback on NTSC or PAL SECAM compliant systems. De-interlacing does not actually affect detection performance, but is a standard process used by pirates to improve the quality of captured video. Video with an aspect ratio of 2.39 is fully captured with an approximately 4: 3 aspect ratio; The upper and lower regions of the video are roughly cropped. Captured video typically indicates a disturbing blink, due to an aliasing effect in the time domain. This blink corresponds to a fast brightness change, which can be filtered out. Flicker cancellation filters are often used by pirates to eliminate this flickering effect. Even if the de-blink filter is not used for the purpose of erasing the watermark, this filter can be very harmful to the time structure of the watermark because it can strongly low pass filter each frame. Finally, the captured movie is compressed to fit the available distribution bandwidth / media / format such as DIVX or other lossy video format. For example, a movie found on a P2P network often has a file size that allows storing a full 100 minute movie on a 700 megabyte CD. This corresponds to an approximate total bit rate of about 800 kbps if 934 kbps, or 128 kbps is maintained for the audio track.

This attack sequence corresponds to the most severe process, which occurs during the lifetime of pirated video that can be found on a peer-to-peer (P2P) network. It also includes, explicitly or implicitly, most of the above mentioned attacks that the watermark must withstand. In addition to the camcorder attack, the watermarking methods and apparatus of the present invention also withstand frame-editing (removal and / or addition) attacks.

The watermarking detection system is called 'blind' (or non-blind) when the detector does not need to access the original content. There is also a so-called semi-blind system, which only requires access to data derived from the original content. Forensic tracking for some applications, such as session-based watermarks for digital cinema, does not explicitly require a blind watermark solution and allows access to the original content, since detection is typically done offline. Because it is. The present invention uses a blind detector but inserts a synchronization bit to synchronize the content at the detector. Semi-blind detectors can also be used with the present invention. If a semi-blind detector is used, synchronization may eventually be performed using data derived from the original content. In this case, no synchronization bit is needed, and the size of the watermark, also called a watermark chip, can be reduced.

In a particular example of a digital cinema application, a minimum payload of 35 bits needs to be inserted into the content. This payload must contain a 16-bit timestamp. If a time stamp is generated 24 hours / day and 366 days / year, every 15 minutes (four times per hour) and the stamp repeats year after year, a 35,136 time stamp is required, which can appear as 16 bits. The remaining 19 bits can be used to indicate the location or serial number for a total of 524,000 possible location / serial numbers.

In addition, all 35-bits are required to be detectable from the 5 minute segment. In other words, only 5 minutes of video should be required to extract forensic marks. In one embodiment, the present invention uses a 64-bit watermark, wherein the watermark chip is repeated every 3:03 minutes. A video watermark chip inserted in 3:03 minutes video at 24 frames per second with one inserted bit per frame has 4392 bits (183 seconds * 24 frames / second = 4392 frames = 4392 bits at 1 bit / frame).

The video watermarking method of the present invention is based on changing the relationship between different characteristics of the content. In particular, in order to encode a bit of information, certain coefficients of the image / video are selected, assigned to different sets, and manipulated in a minimal manner to generate relationships between different sets of characteristic values. The coefficient set has different characteristic values, which generally change in different space-time regions of the video, or after processing the content. In general, the present invention utilizes characteristic values that change in a monotonic manner, for which the attack has a predictable effect since it is easier to ensure a strong relationship in this case. These characteristics will be marked as 'invariant'. The present invention is best practiced using invariant properties, but is not limited thereto and can be practiced using invariant properties. For example, the average luminance value of a frame is considered to be 'invariant' over time: it generally changes in a slow, monotonous manner (except for a boundary shot); Further, attacks such as contrast enhancement will generally take into account the relative order of the luminance values of each frame.

Video content is typically represented by a number of distinct components (or channels) such as RGB (red / green / blue, widely used in computer graphics and color television), YIQ, YUV and YCrCb (used in broadcast and television). YCrCb consists of two main components: luminance (Y) and saturation (also known as CrCb or UV). The amount of brightness or Y-component of the video content indicates the brightness of the content, and the color difference (or saturation) describes the color portion of the video content, which includes hue and saturation information. Tint refers to the color tint of an image. Saturation describes a condition where the output color is independent of changes in the input parameters. The color difference component of YCrCb includes a red (Cr) component of color and blue (Cb). The present invention regards the video content as a plurality of 3D coefficient volumes having a size of W * H * N (W, H are the width and height of the frame in the baseband region or the transform region, respectively, and N is the frame of the video. Number). Each 3D volume corresponds to one component representation of the video content. Watermark information is inserted by making a constraint-based association between certain characteristic values of a selected coefficient set within one or more volumes. However, since the human eye is much less sensitive to changes in overall intensity (luminance) than to changes in color (color difference), the watermark is preferably embedded in a 3D video volume representing the luminance component of the video content. Another advantage of luminance is that it is more invariant to the conversion of video. In the following, the 3D video volume represents a luminance component until otherwise specified, which may represent any component.

In the present invention, the coefficient set may comprise any number of coefficients (from 1 to W * H * N) taken from any position in the content. Each coefficient has a value. Therefore, different characteristic values can be calculated from the coefficient set-some examples are given below. To insert watermark information, multiple associations can be made by changing the coefficient values in the multiple coefficient sets. The association should be understood as one or a set of conditions that, in a non-limiting manner, one or more characteristic values of one or more coefficient sets must be met.

Various types of properties can be defined for each coefficient set. The characteristic is preferably calculated in the baseband region (such as brightness, contrast, brightness, edge, color histogram) or in the transform region (energy in frequency band). Some characteristic values can be calculated equally in the baseband and the transform domain, as in the case of luminance.

One suitable way to insert bits of information is by selecting two sets of coefficients and implementing a pre-defined association between their characteristic values. This association may be, for example, that one characteristic value of the first coefficient set is greater than the corresponding characteristic value of the second coefficient set. However, it is noted that there are some variations in the way to insert bits of information. One way to insert more than one bit of information in two selected coefficient sets is to establish an association between the values of more than one characteristic of the two coefficient sets.

It is also possible to insert information bits using only one coefficient set and by associating the characteristic values of the coefficient set. For example, the characteristic value may be set larger than a certain value, which may be predefined or adaptively calculated from the content. It is also possible to insert more than two bits of information by using one coefficient set, defining four exclusive intervals, and subjecting the characteristic values to fall within a certain interval. Other ways to insert more bits than one include using more than one characteristic value, and associating each characteristic value.

In general, the basic scheme may be generalized for any number of coefficient sets, any number of characteristic values, and any number of associations to be made. This may be advantageous for inserting larger amounts of information, but certain techniques, such as linear programming, may need to be used to ensure that various associations are performed simultaneously with minimal perceptual changes. As noted above, it may be easier to make an association if a constant characteristic value is used.

Many characteristics in 3D video volumes (and coefficient sets) are relatively invariant in a space-time manner and / or before or after processing of content. Examples of invariant properties include:

Coefficients in consecutive frames or in different subbands of the same frame (e.g. wavelet coefficients)

Average luminance value in consecutive frames

Average texture features in consecutive frames

Average edge measurement in consecutive frames

Average color or luminance histogram distribution within successive frames

Energy within a certain frequency range

Any of the above immutable properties in the area defined by the extracted feature point

Watermarking algorithms generally work with secret 'keys', which are known only to inserters or detectors. Using a secret key offers similar advantages as in an encryption system: for example, the details of a watermarking system can generally be known without compromising the security of the system, so that algorithms can be used for peer review and potential It may be released for improvement. Furthermore, the secret of the watermarking system is held in the key, i.e. the user can only insert and / or detect the watermark if the key is known. The key can be more easily hidden and transmitted due to its compact size (typically 128 bits). Symmetric keys are used to pseudo-randomize certain aspects of the algorithm. Typically, this key is used to encrypt the payload after it has been encoded for error correction and detection and expanded to fit the content (eg using a standard encryption algorithm such as DES). For the method of the present invention, this key can also be used to establish an association, which association will be inserted between the characteristic values of two different coefficient sets. Therefore, this association is considered 'pre-defined' because this association is fixed for a given private key. If there is more than one pre-defined association for embedding a watermark, this key can also be used to randomly select a precise association for a given bit of information and for a given set of coefficients.

The selected coefficient set generally corresponds to a 'zone', which should be understood as a coefficient set located within the same area of content. The coefficient zone may correspond to the space-time zone of the content, which is the same as for baseband coefficients and wavelet coefficients, but need not necessarily be the case. For example, the 3D Fourier transform coefficients of the content do not correspond to either spatial or temporal zones but to similar frequency zones.

For example, a coefficient set may correspond to a zone, which may consist of all the coefficients within a given spatial region for one frame. To encode the information bits, two zones from two consecutive frames are selected and their corresponding coefficient values are changed to effect an association between certain characteristics. As described in more detail below, it is noted that it may not necessarily be necessary to change the coefficient value if the desired association already exists.

Again for another example, four wavelet coefficients (LL, LH, HL, and corresponding to four subbands for each position and each component (channel) at each resolution level for each frame using wavelet transform HH) is present. The coefficient set may contain only one coefficient in one of four subbands. Suppose C1, C2, C3 and C4 are four coefficients located at the same position, channel and resolution level but within each of the four subbands. One way to embed a watermark is to establish an association between C2 and C3, corresponding to the coefficients in each of the HL and LH subbands. One example of an association is that C2 is greater than C3. Another method for embedding a watermark is to perform an association between C1-C4 in the frame and the corresponding coefficient in the continuous frame. A variation on this principle is by inserting an association for only one type of coefficient, which must be larger than the pre-calculated value. For example, for all positions in a frame at a constant resolution level, it is possible to enforce that the value of the coefficient LL is greater than the pre-calculated value. In the above example, the characteristic value is the value of the wavelet coefficient itself.

It is essential that the detection side can identify the same, or nearly the same set of coefficients as on the watermarking side. Otherwise, an incorrect coefficient is chosen and the measured characteristic value will be errored. Identifying the correct coefficient is generally not a problem if the content has been mildly processed prior to detection, in which case the position of the coefficient does not change (regardless of whether it is spatial or transformed). However, if this process changes the geometric or temporal structure, the coefficient is likely to change position, as is generally the case during a camcorder attack.

If there is a change in the temporal structure of the content, the user can resynchronize the content using a non-blind or semi-blind scheme. Different methods are available in the prior art for this purpose. If detection should be done blindly (ie, without access to any data derived from the original content), inserting a synchronization bit with a predictable value into the content to be used by the detector to resynchronize the content. It is possible. This approach will be described in more detail below.

In order to ensure robustness against changes in the geometric structure of the content, a synchronization / registration method known in the art can be used, which matches the location in the changed content with the corresponding location in the original content. Restore it. Changes in the geometric structure of the content may occur when the original content or some data derived therefrom is available (e.g., some characteristic information or thumbnail of the original content), e.g., rotation, scaling and / or content. Or occurs after cropping.

In the case of blind detection, one possibility is to use a very low spatial frequency. For a video frame or image, one section of the coefficient may correspond to a complete video frame, half or one quarter of a frame. In this case, most of the coefficients will be chosen correctly (all coefficients if the zone corresponds to a complete video frame), and detection is generally robust even if some coefficients are assigned to the wrong set.

Another way to be inherently robust to changes in geometric structure is to actually use a region containing only one coefficient, and one coefficient in one frame and one coefficient in the corresponding position in the next frame. Is to establish an association between them. If the same association is made for all coefficients in two frames, the user can easily see that the detection is inherently robust to geometric distortion. An associated way to ensure robustness to changes in geometric structure is to create an association between different wavelet coefficients at certain locations within different subbands. For example, there are four coefficients corresponding to the four subbands LL, LH, HL and HH for each resolution level, each position and component (channel) in the wavelet transform. The same association between two coefficients for all positions in the frame can be done at a constant resolution level to insert watermark bits to enhance watermark robustness. On the detection side, the number of times an association is observed is an indicator of which bits have been inserted.

Another way to ensure robustness to changes in geometry is to use feature points that are invariant to changes in geometry. Here, constant means when the same point is found on the original content and the changed content, using a certain algorithm to extract the feature point of the video or image. Different methods are known in the art for this purpose. Such feature points can be used to delimit coefficients in the baseband and / or transform regions. For example, three adjacent feature points define an inner zone, which may correspond to a coefficient set. In addition, three adjacent feature points may be used to define the subzones, each subzone corresponding to a coefficient set.

Another way to be inherently robust to changes in geometric structure is to make an association between the value of the global property of all the coefficients in one frame and the value of the same global property of all the coefficients in the second frame. It is assumed that these global properties are invariant to changes in geometry. An example of such a global characteristic is the average luminance value of one image frame.

A non-limiting example algorithm for inserting bits by constraining between feature values of two consecutive frames of video is as follows:

For each frame, which is a JPEG2000-compressed image within the frame sequence of the video (F1, F2, ... Fn):

a) Select a zone, consisting of N coefficients at resolution level (L). Coefficients may belong to one or more subbands such as LL, LH, HL, and HH. The zone may be of any but fixed shape (eg, rectangular) or may vary according to the original image content, eg using feature points for additional stability of the zone when facing a geometric attack as described above.

b) Determine the relevant global characteristics for the zone. The global characteristic may be an average luminance value of the zone, an average texture feature measurement, an average edge measurement, or an average histogram distribution. P is the value of this global property.

To insert bit strings {b1, b2, ... bm}:

a) bi (1≤i≤m) the case of _{0, P (F 2 * i} + 1)> P (F 2 * i) is such that (F by at least _two methods, if necessary only) _{i *} and F ₂ Change _{i + 1} .

b) bi (1≤i≤m) is the case of _{1, P (F 2 * i} + 1) <P (F 2 * i) is such that (only if necessary) to the minimum system F _{2 i *} and F ₂ Change _{i + 1} .

This algorithm can be extended to insert multiple bits per frame by inserting associations between several characteristic values of two frames.

For watermark detection:

a) Synchronize the captured video in the time domain. This can be done using synchronization bits, non-blind or semi-blind schemes.

b) Select a zone consisting of N coefficients at level (L). As with insertion, the zone may be of a fixed shape.

c) Calculate the relevant global characteristics for the zone. P 'is the value of the global property of the zone.

d) Bit 0 is detected when P '(F _{2 * i + 1} )>P' (F _{2 * i} ).

e) Bit 1 is detected when P '(F _{2 * i + 1} ) <P' (F _{2 * i} ).

Watermarking in the present invention is divided into three steps: payload generation, coefficient selection, and coefficient modification. Three steps are described in detail below as exemplary embodiments of the present invention. Quite a few variations are possible for each of these steps, and these steps and descriptions are not intended to be limiting.

Referring now to FIG. 2, this diagram is a flow chart illustrating a payload generation step of watermarking, where a secret key is retrieved or received at step 205. Information identifying the time stamp and a number identifying the device's serial number or location are retrieved or received at step 210. The payload is generated at step 215. The payload for digital cinema applications is at least 35 bits and 64 bits in the preferred embodiment of the present invention. The payload is then encoded for error correction and detection, eg, using BCH coding at step 220. The encoded payload is optionally duplicated in step 225. Optionally, a synchronization bit is then generated based on the key at 230. The synchronization bit is generated and used when using blind detection. This synchronization bit can also be generated and used when using semi-blind and non-blind detection schemes. If a synchronization bit has been generated, this synchronization bit is assembled with the sequence at step 235. The sequence is inserted into the payload in step 240 and the entire payload is then encrypted in step 245.

Payload generation involves translating the specific information to be inserted into a bit string, which is called a "payload." This payload to be inserted is then extended through the addition of error correction and detection capabilities, potential iterations along the synchronization sequence, encryption and available space. An example sequence of operations for payload generation is:

1. Translate the "information" to be inserted into the "original payload". Convert the information (timestamp, projector ID, etc.) into payload. One example has been provided above to generate a 35 bit payload for a digital cinema application. In an exemplary embodiment of the invention, the payload has 64 bits. Calculate the "encoded payload" from the original payload, the encoded payload includes error correction and detection capability. Various error correction codes / methods / methods may be used. For example, BCH coding. The BCH codes 64 and 127 can correct up to 10 errors in the received bit stream (ie, approximately 7.87% error correction rate). However, if the encoded payload is repeated several times, a significant number of errors can be corrected due to redundancy. In an exemplary embodiment of the present invention, the 127-bit repeated encoding payload is repeated 12 times, and it is possible to correct up to 30% error in the individual bits inserted in each frame.

2. Depending on the space available, duplicate the encoded payload to obtain "Duplicate encoded payload." In the present invention, duplicate each of the encoded bits 12 times for a total of 127 (BCH coding) * 12 = 1524 bits.

3. Using the key, encrypt the duplicated encoded payload to obtain the "encrypted payload"; The encrypted payload is typically the same size as the duplicated encoded payload.

4. Generate synchronization bits (optionally before encryption) and insert them in various places within the repeated encoded payload; The resulting sequence is a video watermark payload. For example, calculate a fixed synchronization sequence with 2868 bits. This sequence is divided into one global synchronization unit of 996 bits (as the header of the watermark chip) and 12 local synchronization units of 156 bits (for the header of each payload). In this example, multiple bits are used as synchronization bits. When using a non-blind method at the detector, where the original content is used to synchronize the test content in time, the synchronization bit is still very useful for locally adjusting the registration. In other words, the synchronization bits otherwise take up space that can be used for additional redundancy of information and thus increase the robustness to individual bit errors. However, synchronization bits increase the precision and quality of the extracted information, which results in fewer individual bit errors. The number of inserted synchronization bits is thus set as the best compromise that causes the minimum number of errors within the 127 encoded bits.

5. Assemble the watermark chip by concatenating the following bits in sequence:

■ Global synchronization (996 bit) synchronization unit.

First 127 bits of the encrypted payload, where the first local synchronization unit (156 bits)

The second 127 bits of the encrypted payload, where the second local synchronization unit (156 bits)

■ ...

Last 127 bits of encrypted payload, last local synchronization unit (156 bits)

Watermark chips (eg 4392 bits) are typically a few orders of magnitude larger than the original payload (eg 64 bits). This allows recovery from errors that occur during transmission on the noisy channel.

Referring now to FIG. 3, this diagram is a flowchart illustrating the selection of coefficients for watermarking, where a key is retrieved or received at step 305. The payload (encrypted, synchronized, duplicated and encoded) is retrieved at step 310. The coefficients are then separated into disjoint sets based on the keys in step 315. Based on the payload bit and the key, a constraint between the characteristic values is determined at 320.

The choice of coefficients can occur in either the baseband or the transform domain. Coefficients are selected within the transform domain and grouped into two disjoint sets C1 and C2. The key is used to randomize the coefficient selection. The characteristic values, P (C1) and P (C2), for each of the two sets are identified, which are generally identified to be invariant for C1 and C2. Various such characteristics can be identified, such as mean values (eg, luminance), maximum values, and entropy.

The key and bit to be inserted are used to establish a relationship between the characteristic values of C1 and C2 such as P (C1)> P (C2). This is called a constraint decision. For additional strength, a positive value 'r' can be used, such that P (C1)> P (C2) + r. The association may already be appropriate, in which case the coefficients do not need to be changed. In the worst case, P (C2) can be significantly larger than P (C1), for example, if P (C2) is already greater than P (C1) + t, t is a predetermined value or perceptual Depending on the model, changing the coefficients in this case is not worth it because it can cause perceptual damage. In most cases, however, P (C1) is P'1 = P (C1) + p1 and P (C2) is P'2 = P (C2) -p2 (p1 and p2 are positive numbers), Let P'1> P'2 + r.

Referring now to FIG. 4, this diagram is a flow chart illustrating a coefficient changing step of watermarking in step 405, where a disjoint coefficient set is received or retrieved. The characteristic value for the disjoint coefficient set is measured at step 410. The characteristic value is tested in step 415 to determine the distance between these values, which is a measure of strength. If the characteristic value is within the critical distance t, step 420 is reached, since no coefficient change is necessary. If the characteristic value is greater than the threshold distance r, an additional test is performed in step 425 to determine if the characteristic value is within a certain maximum distance allowed to perform the coefficient change. If the characteristic value is within the maximum distance, the coefficient is changed at step 435 to satisfy the constraint association. If the characteristic value is not within the maximum distance, the coefficient does not change as described by step 430.

The watermarking method of the present invention is " adaptive " to the original content, since a change to the content ensures that the minimum or bit value will be detected correctly. Spread spectrum watermarking methods may also be adaptive to the original content but in a different manner. Spread-spectrum watermarking methods take the original content into account to control the change, so as not to cause perceptual damage. This is conceptually different from the method of the present invention, which can determine not to insert any changes in any area of the content at all, not because these changes can be perceived, but because the desired association already exists or This is because the desired association cannot be established without significantly deteriorating the content. As shown below, the method of the present invention, however, can be adapted to ensure that the bits will be detected correctly and to minimize perceptual damage.

The method of the present invention results in a greater amount of distortion than the spread spectrum method for the same distortion and bit rate since it causes the least amount of distortion to ensure that the bits are inserted strongly and gives up if the distortion is too severe.

In the baseband region, one embodiment of the present invention separates the pixels within each frame into top and bottom. The brightness of the top / bottom is increased or decreased depending on the bit to be inserted. Each frame is divided into four rectangles in the space area from the center point. Dividing the frame into four rectangles allows storage up to 4 bits per frame. This method includes:

Grouping pixel values into the top and bottom of the frame to form two sets of coefficients C1 and C2.

Measuring luminance, that is, P (C1) is the average of all coefficients in C1 and the same for C2.

Changing the pixel value only if necessary to set constraints, eg P (C1)> P (C2) + r, and in a minimal manner, where r is generally a positive value.

In this embodiment of the invention, the watermarking insertion module only accesses the lowest resolution coefficient of the wavelet transform of the image. For video frames with a pixel size of 2048 (width) x 856 (height) pixels, 64 × 28 = 1728 coefficients, or 1728, for each subband (ie, LL, LH, HL, and HH) at resolution level 5 * 4 = 6912 exists. Only these coefficients, a subset of these coefficients, are used for video watermark embedding. Two non-limiting methods are described below using the coefficient group selected within the frame.

In the first method, only LL coefficients (also called approximation coefficients) are used for video watermark embedding. The LL coefficient matrix 64x28 is divided into four tiles / parts from the 32x14 center points C1, C2, C3 and C4. Depending on the bits and keys to be inserted, a constant association is created between the respective coefficients of four parts, LLa (top left section), LLb (top right), LLc (bottom right), and LLd (bottom left), each part It is created by increasing / decreasing the coefficient of, so that certain constraints are met. Each of the four rectangular tiles / portions may have between 286 and 1728 coefficients for each of the three color channels. To smooth the watermark (and to limit the visibility of the watermark) in the transition between zones LLa to LLd, the transition zone can be left non-watermarked or watermarked with lower intensity.

An example of a constraint may be: P (C1) + P (C2)> P (C3) + P (C4). For linear characteristics such as average luminance, this equation can be written as P (C1 ∪ C2)> P (C3 ∪ C4) if it has only two zones instead of just four, but this is equal to the maximum value of all coefficients. It is generally not true about the linear characteristic. There are several different possible constraints depending on the bits to be inserted and the keys used.

One advantage of separating the coefficients into four tiles is that in addition to allowing for constraints, it also allows the use of very low spatial frequencies. As described above, this frequency is robust to geometric attacks, while allowing for storing a higher number of bits than a method that only considers the global characteristics of the frame.

In the second method the coefficients LH and HL are used for video watermark embedding. There are various ways to manipulate these coefficients to insert constraints. Bits are inserted by inserting constraints between the coefficients LH and HL at the lowest resolution level. For example, a constraint may exist for all x, y such that the coefficient LH (x, y, f)> HL (x, y, f) in frame f. As these constraints are often too strong to actually apply literally, the coefficients can be manipulated so that the association is applied globally. for example,

So that it can exist.

It should be noted that the second association is not linear and allows fine grain but more complex coefficient insertion. This allows the change to be counted so that if there is, the area more sensitive to the change does not change that much.

It should be noted that in this method instead of changing the pixel values, a relatively small number of coefficients (64x28 LL coefficients) are changed to change the luminance of the frame. This is a huge advantage for watermark embedding, especially in applications that have limited computational resources and require cost-effective and real-time watermarking capabilities.

Several more methods can be anticipated according to the coefficient set, which can utilize coefficients within one frame or coefficients from consecutive frames, specified characteristics, association types to implement, and the like. In general, the most viable method will usually use a set of coefficients with invariant characteristics in the sense that the alignment of characteristic values is generally preserved after a change to the content.

To change the coefficients, in one embodiment the present invention uses two sets of coefficients C1 = {c11, ..., c1N} and C2 = {c21, ..., c2N} and changes their values. do. The values of the coefficients cij are denoted by v (cij) and v '(cij) before and after the change, respectively.

As discussed above, more than two coefficient sets can be used for more demanding associations. It is also possible to use only one coefficient set. Without loss of generality, it may be desirable to set the association such that P (C1)> P (C2) + r, where r is a value that adjusts the strength of the association.

If the function P is at its maximum, for example, manipulate the strongest coefficients C1 and C2 only in the following manner to minimize the change.

In the above, the function P is strongly non-linear, ie the property does not vary smoothly as a function of the coefficient value. This method is advantageous because it allows the insertion of bits by changing only one coefficient per set (although the change may have to be strong).

An extension of this 'maximum' method, which can make it more powerful, replaces not only the maximum but also the N strongest values (N is typically significantly smaller than the size of the coefficient set), giving the opportunity to correctly decode the association after manipulation on the content. Maximize It is understood that several other variations are possible with this technique.

On the other hand, if the function P is a linear characteristic of the coefficients (e.g., mean), the change can be distributed randomly over all coefficients in each set. For example, to establish an association

It is assumed that it is desirable to change the mean value of the coefficients so that the change can be equally distributed over each coefficient (positive for coefficients belonging to C1 and negative for coefficients belonging to C2),

이며, 이 경우 계수는 변경될 필요가 없 다., And the same for c2j. If the association is already maintained

In this case, the coefficient does not need to be changed.

As described above, the basic method can be extended to incorporate more associations by using different characteristics. For example, consider the 'maximum' and 'average' methods together to have four associative combinations between the two sets, allowing encoding of two bits. At this time, the following association may take place:

In addition, as described above, only one set of coefficients may need to be used, in which case the association is established for a fixed or predetermined value. For example, the association may be made such that the maximum or average C1 is higher than a certain value. In another case, the key can be used to pseudo-randomly select to make 'max' or 'average' association depending on the key, which significantly improves the security of the algorithm.

The approach described above can incorporate a masking (perceptual) model, which allows to distribute the intensity of the watermark within each area of the image causing a minimal perceptual impact of the watermark. This model can also determine if manipulation is possible to make the association without perceptual damage. The following describes a non-limiting way to integrate a watermarking model for video content in the context of real-time watermarking in a digital cinema projector.

There are two main masking effects for images: texture masking and brightness masking. Furthermore, the video benefits from the third masking effect: temporal masking.

In some applications, such as digital cinema, which have limited computational resources but require real-time watermarking, for example at resolution level 5, it may be desirable to simply use the subband coefficients (LL, LH, HL and HH) of the lowest resolution level. have. The final three types of coefficients are potential indicators of texture, while LL is an indicator of brightness. However, the corresponding resolution is low and the texture masking effect is not significant at this resolution. To illustrate this, let's compare a video frame at full resolution with the same video frame reconstructed from the coefficients at resolution level 5. See FIG. 5. Most of the texture seems to be lost at this resolution. Therefore, the LH, HL and HH subband coefficients for level 5 are coarse indicators of the texture and will not be used for texture masking measurements.

However, temporal masking can still be estimated with fairly good precision, since motion can be applied to a rather large area of video, which is therefore low frequency. Temporal masking can be measured by subtracting the coefficient of the previous frame from the coefficient of the current frame. C (f, c, l, b, x, y) is frame f, channel (i.e. color component) c, resolution level l, subband b, coefficients LL, LH, HL and B = 0 to 3), position (x, y) for HH. Thus, the sum of the absolute differences between coefficients of the same type on two consecutive frames is a valid measure of temporal change:

For a given frame f, resolution level l = 5, T (f, c, l, b, x, y) is measured for all positions (x, y) and for each color of the color channel (Typically there are three color channels / components). If there are several channels, it may be advantageous to take the average value of T (f, c, l, b, x, y) over all channels. At this time, for each position (x, y), the value of T (f, c, l, b, x, y) is compared with the threshold t, and the coefficient at this position is only if the value is higher than t. Is changed. Experimentally, a good value for t is 30. When the coefficient changes, a certain amount of change is made as a function of brightness, as is known in the art.

6 is a block diagram of watermarking in a D-cinema server (media block). The media block 600 includes a module that can be implemented in hardware, software, firmware, etc. to perform watermarking, including at least watermark generation and watermark insertion. Module 605 performs watermark generation, including payload generation. The encoded watermark 610 is then forwarded to the watermark embedding module 615, which receives coefficients of the image from the J2K decoder 625 and then selects and changes the wavelet coefficients 620, and finally, The modified coefficients are returned to the J2K decoder 625.

As described above, the watermark generation module generates a payload, which is a bit string to be inserted directly. The watermark embedding module takes the payload as input, receives the wavelet coefficients of the image from the J2K decoder, selects and changes the coefficients, and finally returns the changed coefficients to the J2K decoder. The J2K decoder continues to decode the J2K image and output the decompressed image. As an alternative design, the watermark generation module and / or watermark embedding module may be integrated into the J2K decoder.

The watermark generation module may be called periodically (eg, every 5 minutes) to update the time stamp in the payload. Therefore, it may be referred to as "offline", that is, the watermark payload may be generated in advance in the D-cinema server. In any case, the computational requirements of this module are relatively low. However, watermark embedding should be done in real time and the performance of this module is important.

Video watermark embedding can be done with varying levels of complexity in a way that original content can be considered. More complexity may mean additional fidelity for a given fidelity level or more fidelity for the same fidelity level. However, this involves additional costs in terms of computation.

Before estimating the number of operations required for embedding a video watermark, it is noted that any of the following basic calculation steps are considered as one operation:

· Bit shift of coefficient

Addition or subtraction of two coefficients

Multiplication of two integers

Comparison of two coefficients

Access values in the lookup table

In the following example, C (f, c, l, b, x, y) and C '(f, c, l, b, x, y) are respectively used for the color channel c for frame f. Original coefficient and water at position (x (width)), y (height) for frequency band b (0: LL, 1: LH, 2: HL, 3: HH) at wavelet transform level l Marked coefficient. Furthermore, it is assumed that N is the number of coefficients at the lowest resolution level that needs to be changed.

For the sake of simplicity, it is assumed in the following that the coefficient value is increased during video watermark embedding. However, it is noted that the addition in the equation can be equally replaced by the subtraction.

If each coefficient varies by the same amount, then there is only one action per coefficient:

Here, the value a is a constant. One additional comparison operation may be required to check for overflow of the changed coefficients. Therefore, the total calculation requirement is 2 * N.

However, the above is not an effective method. In fact, if the constant value a is too large, the watermark will be visible. Therefore, the value a should be conservative, i.e. small enough so that the watermark never causes a visible defect, on the other hand, if the video watermark is too conservative, it may not withstand severe attack. The LL subband coefficients correspond to local luminance, while the LH, HL and HH coefficients correspond to image changes, or “energy”. It is well known that the human eye is less sensitive to changes in luminance in bright areas (stronger LL coefficients). It is also less sensitive to changes in areas with strong changes, depending on the coefficients LH, HL and HH, depending on the direction of change. This must however be carefully considered: LH and HL coefficients can counter perceptually significant changes, such as edges, which must be carefully manipulated.

Nevertheless, it may be advantageous to make a change proportional to the coefficient, at least for the coefficients LL and HH. Simple proportional changes can be made by copying the original coefficients, by bit-shifting the copied coefficients, and by adding or subtracting the bit-shifted coefficients, for example

Typical values for n are 7 or 8. For n = 7 or 8, the coefficient is changed by 1/128 or 1/256 of its original size. For example, for an image having an average brightness of 128 on a size from 0 to 255, the effect of the coefficient change is a change in brightness of one. This change typically does not produce visible defects.

There are two operations per coefficient. In possible overflow checking, the total calculation requirement is 3 * N, where N is the number of manipulated coefficients.

It is also noted that it is possible to impose a minimum change a to ensure that the watermark is inserted sufficiently strong for a frame with very low luminance. In this case there are three actions per coefficient:

In addition, the following perceptual features can be used to make adaptive changes in coefficients:

Temporal situation. Temporal masking is related to temporal activity, which is best estimated by using coefficients in previous, current and subsequent frames, and the present invention is directed to determining the temporal activity of the preceding and current frames. Use only coefficients. High temporal activity allows for stronger watermarks. The estimated computational complexity for temporal modeling is about four.

Texture situation. For each coefficient C (f, c, b, l, x, y), in order to model texture and flatness, the K additional correspondence coefficients within the other subbands have an estimated complexity of 4K ² motion. Can be used.

Brightness situation. A lookup table may be used to determine weights according to luminance in coefficients C (f, c, b, l, x, y). The estimated operation is B, where B is the number of bits representing the luminance value.

All perceptual features can be weighted and balanced to determine changes in coefficients:

Where W is a weight that combines all perceptual features.

A rough estimate of the watermark embedding complexity, for convenience the complexity is estimated in terms of the number of operations as described above. It should be noted that the number of operations may vary depending on the exact manner in which the operations are defined, the watermarking and masking procedures implemented, and the like. Nevertheless, the method of the present invention is robust when a relatively small number of coefficients that need to be accessed by the method of the present invention (about 1 / 1000th of the image size) and a relatively small number of operations per coefficient are provided. It can be concluded that the calculation is feasible.

Referring now to FIG. 7, watermark detection generally consists of four steps: video preparation 705, extraction and calculation of feature values 710, detection of bit values 715, and embedded (watermarks). Decoding 720 information. A test is performed at 725 to determine if the watermark information has been successfully decoded. The process is complete when the watermark information has been successfully decoded. If the watermark information was not successfully decoded, the above process may be repeated.

Video preparation itself includes resizing or resampling of video content, synchronization and filtering of video content:

Resampling of the transformed (distorted) video may have to be done if the frame rate is different at insertion and detection. This is often the case, while the frame rate for insertion is 24, while detection may be, for example, 25 (PAL SECAM) or 29.97 (NTSC). Resampling is performed using linear interpolation. The output is resampled video.

Filtering resampled video, typically with a high-pass time filter to reduce noise due to cover content or to emphasize the watermark. The output is filtered video.

Synchronization of the filtered video can be done with the original content using various methods as described above, or by cross-correlation with the synchronization bits when the synchronization bits are inserted into the video content. Typically, if very low spatial frequencies are used, only temporal registration should be done. The local synchronization unit and optionally assembled global synchronization unit are used to determine the starting point of the watermark sequence. Cross-correlation is performed between the filtered video and known sync bits. There is typically a strong peak in the cross-correlation function for the corresponding movement of the video. Referring now to FIG. 8, the local synchronization process retrieves the next local synchronization sequence / unit at 805. The video portion corresponding to the watermark chip is then retrieved at 810. The video portion and local synchronization sequence / unit are cross-correlated at 815. The peak value of the cross-correlated characteristic value P1 is located at 820 and the peak value of the characteristic value P2 is located at 825. A test is made at 830 to determine whether the characteristic value P1 is greater than the characteristic value P2 plus a predetermined value or whether the characteristic value P1 is smaller than the characteristic value P2 plus a predetermined value. If the test result is negative, the video portion is rejected at 835. If the test result is positive, the video portion is retained at 840. Additional testing is performed at 845 to determine if the end of the video has been reached. When the end of the video is reached, a local synchronization process is done. If the end of the video is not reached, the local synchronization process is repeated. 9 shows a cross-correlation function (actually a low pass filtered version of size) with two peaks representing the starting point of two consecutive watermark chips. Once the start point of the watermark chip is located, a local synchronization unit located at the start point of each payload is used for slight reordering of the video at regular intervals. In turn, each of the twelve local synchronization units is cross-correlated with the filtered video in a small window around the expected position. If a relatively strong correlation peak is found (as determined by the difference between the highest peak and the second highest peak), the adjacent filtered video is maintained for the next step, otherwise discarded. The principle is that a stronger correlation peak is an indicator that the filtered video is synchronized more precisely. The output of this step is synchronized video.

The output of the three stages of video preparation will be marked as 'processed video' in the following. The processed video is a data set, which is calculated from the received video to facilitate extraction / calculation of feature values, the next step in watermark detection.

In one embodiment of watermark embedding as described previously, the average luminance of each of the four quadrants is calculated for each frame. The characteristic value forms a vector of frames x4. For wavelet watermark embedding using LL subband watermarking, the feature value may be extracted from the baseband representation or wavelet of the received video. In both cases, processed video of the size of the number of frames x 4 is obtained. In all of the above manner, the frame is separated into four parts / tiles from the center point. This center point can be automatically set at the center point of the frame-as in the original video-but naturally has some offset within the camcorder captured video.

Extracting and calculating feature values for wavelet watermark embedding using the LH and HL subbands operates slightly differently. Changing the LH coefficient produces a stripe with a frequency that can be precisely determined (stripes are equally spaced horizontal lines in the baseband video), at least before any attack in the watermarked video. This stripe is not visible when the watermark energy is adjusted using the masking model as described above. The user can thus calculate the converted video by measuring the energy within that frequency (eg, using a Fourier transform). However, during the camcorder attack and subsequent cropping of the video, the associated frequency can be shifted, and its energy spreads on adjacent frequencies. Therefore, energy signals for every frame are collected within a 5x5 window around the relevant frequency. Each of these 25 signals is tested for cross-correlation peaks with synchronization bit sequences, and one with the highest peak is output as a characteristic value.

In the watermark detection step, the feature value is calculated corresponding to the way in which the watermark is inserted. The watermark is:

■ characteristic values of continuous frames;

■ one characteristic value and a predetermined value of the region of the frame;

■ Characteristic values of one section of the frame and another section of the same frame

■ Characteristic values of one zone of the frame and the corresponding zone of successive frames

Intercalation and / or among them, by inserting at least a subsequent association.

Since the characteristic value can also be the coefficient value itself, the watermark is:

One coefficient value and a predetermined value in the video volume;

One coefficient value in one subband of the frame and the remaining coefficient values in the corresponding position and subband in the continuous frame;

One count value in another subband of the same frame as one count value in one subband of the frame;

Intercalation and / or among them, by inserting at least a subsequent association.

The characteristic value can be calculated in the baseband and / or the transform domain. Similar to watermark embedding, multiple bits may be detected between and / or among multiple association values.

The first and second steps of watermark detection can be interchanged in sequence. For convenience, it is advantageous if it is possible to calculate the characteristic value first, because this causes data compaction (i.e., reducing the total image data of each frame to several values per frame). The mark can be tailored to a form that can be read more easily. However, it may not always be possible to perform the calculation of the characteristic value first, due to severe distortion of the video, in particular geometric distortion.

The third step receives the characteristic value as an input and outputs the most likely bit value for each of the 127 encoded bits. The feature value may correspond to multiple insertions of each of the 127 encoded bits. In an example according to the principles of the present invention, where each bit is inserted at 12 different positions, there may be up to 12 inserts, but there may be fewer inserts if a given payload unit is discarded due to poor local synchronization. .

Referring now to FIG. 10, a disjoint coefficient set is searched for the next encoded bit at 1005. At 1010, a relevant characteristic value is calculated for the disjoint coefficient set. At 1015, the most likely bit value is determined from the computed characteristic value. The test is performed at 1020 to determine if there are any more encoded bits. The process is repeated if there are any more encoded bits. An exemplary cumulative signal is shown in FIG. 11.

Each bit of the encoded payload is expanded, encrypted and inserted at multiple locations in the content. For each extended bit, as described above, insertion is typically done by setting a constraint between the characteristic values of the two sets of coefficients C1 and C2, eg, P (C1)> P (C2). Assuming there are N such extended bits and thus N such inserted constraints:

Bit = 1 if P (C1i)> P (C2i) for each i, where 1≤i≤N

Bit = 0 if P (C1i) <P (C2i) for each i, where 1≤i≤N

In general, due to channel noise or initial impossibility in establishing an association, not all associations will necessarily necessarily match the inserted bits. The simplest approach to solving this problem is to take a "vote majority" vote. That is, to select the bits where the corresponding association between coefficients is most often observed,

Bit = 1 if the number of cases where P (C1i) <P (C2i) (1≤i≤N) is greater than N / 2

Otherwise, Bit = 0

This approach does not help solve the case where N is even, bit = 1 and the number of associations for bit = 0 is the same. Furthermore, this approach does not make full use of the information of P (C1), P (C2), and possibly other information, which can increase the likelihood of accurately determining the association. A more refined approach consists in estimating the probability that the inserted bit values are 1 and 0, respectively, if observations of the characteristic values P (C1i) and P (C2i) are provided. The individually estimated probabilities are combined using a probabilistic approach, and the decision is made based on the maximum-likelihood (ML) criterion, where the most likely bit is selected. Other criteria are possible, such as the Neyman-Pearson rule.

Using the ML rule, where the most likely bit is selected, the decision is made only based on the characteristic value. The ML rule is:

Indicates.

Using Baye's rule, and assuming each bit is equally probable, it is:

Can be rewritten.

Since the bits are expanded at different pseudo-random locations in the content, it can be assumed that the characteristic values are relatively independent. In other words,

The algorithm is

Take

In order to implement this equation, the equations Prob (P (C1i), P (C2i); bit = 1) and Prob (P (C1i), P (C2i); bit = 0) need to be derived. This equation will depend on the characteristics of the channel. The general technique consists of collecting enough data to estimate this function. Some a priori knowledge, or premise for a probabilistic model (eg, coefficients or noise follow a Gaussian distribution) can be used.

Considering a very specific case where the log of the probability is proportional to the difference between P (C1i) and P (C2i), symmetrically for bit 1 and bit 0:

The rule is:

It becomes

The rules derived for this particular case correspond to simple correlations, as used in spread spectrum systems. This rule, however, is suboptimal because in general the probability will not change in a logarithmic way. This is one reason why the method of the present invention can be seen more generally and more effectively than the spread spectrum based method.

In fact, depending on the original content value, it turns out that due to the particular way constraints are inserted, the probability is generally not a monotonically increasing function. To illustrate this, the following simulation was performed, where the estimates of the bit values were compared based on the observation of the received signal, respectively, for the association-based approach and the classical spread spectrum approach of the present invention.

The original content Gaussian noise (X) was generated. A binary watermark W was taken from [-1, +1] and added to this signal. Binary watermarks were first added according to the constraint-based concept in the following manner:

Values (a1 = 0.5, a2 = -0.5, r = 0.3) were selected. This resulted in a PSNR of -15 dB.

A spread-spectrum watermark was then added to the generated signal in the following manner:

Y2 = X + a * W

The parameter 'a' was adjusted to cause the same -15dB PSNR.

The same noise vector N was added to the two signals Y1 and Y2 to obtain two received signals R1 = Y1 + N and R2 = Y2 + N. Noise also had a PSNR of -10dB for the original content. For the two received contents R1 and R2, the probability that the inserted bit is '1' was estimated when the received signal value was provided. The results are shown in the graph shown in FIG. The difference is significant: As expected, for spread-spectrum insertion, the estimated probability that the bit is 1 increases linearly with the received signal value. However, in the association-based approach of the present invention, the estimated probability has a very specific shape that undergoes a maximum after a minimum. This shape can be explained as follows.

When the cover content has a high or low value, it is logical that it is most likely not used for insertion, so that the received signal is not correlated with the bit.

Estimates are the most reliable at -0.5 and +0.5, which are the minimum / maximum values at which watermarks are inserted.

Therefore, it can be concluded that accurate probability estimates are of great importance for the proper operation of the method of the present invention.

In the final step, once the 127 bit values of the encoded payload are estimated, 64 bit payloads can be decoded using the BCH decoder. With this code, up to 10 errors can be detected from the estimated encoded payload. As described above, this payload contains various information for forensic tracking, such as location / projector identifiers and timestamps in digital cinema applications. This information is extracted from the decoded payload and allows for a wide range of uses, such as forensic tracking of potential frauds that have occurred.

In case of failure at the final stage (ie no valid watermark information is decoded), the above four stages will be followed by each stage until the watermark information has been successfully decoded or the maximum number of such attempts has been reached. Can be repeated with different strategies (eg, optimized synchronization and registration for video in the first step).

It should be understood that the present invention may be implemented in various forms, for example, in a server or mobile device, such as hardware (eg, ASIC chip), software, firmware, dedicated processor, or a combination thereof. Preferably, the present invention is implemented in a combination of hardware and software. Moreover, the software is preferably implemented as an application explicitly implemented on a program storage device. The application can be uploaded to and executed by a machine that includes any suitable architecture. Preferably, the machine is implemented on a computer platform having hardware such as one or more central processing units (CPUs), random access memory (RAM), and input / output (I / O) interface (s). The computer platform also includes an operating system and microinstruction code. The various processes and functions described herein may be part of microinstruction code or part of an application program (or a combination thereof) that is executed through an operating system. In addition, various other parallel devices may be connected to the computer platform, such as additional data storage devices and printing devices.

Since some of the structural system components and method steps shown in the accompanying drawings are preferably implemented in software, it is additionally that the actual connection between system components (or process steps) may differ depending on how the invention is programmed. It must be understood. Given the teachings herein, one of ordinary skill in the art would be able to anticipate such and similar implementations or configurations of the invention.

The present invention is available for watermarking video content, in particular for embedding and detecting watermarks in digital cinema applications.

Claims (44)

A method for embedding a watermark by performing an association between feature values of a selected set of coefficients in a video volume, the method comprising: Generating a payload; Selecting a coefficient; Changing a coefficient, wherein the changed coefficient replaces the selected coefficient; And Embedding the watermark in the video volume Including a watermark. According to claim 1, The characteristic is a measurable characteristic of any portion in the video volume. According to claim 1, Wherein said characteristic value is calculated by one of averaging, determining a maximum, and determining a minimum. According to claim 1, A coefficient change is performed in one of a spatial area and a transform area to effect the association between the feature values. According to claim 1, And a bit of the payload is inserted by performing an association between or among at least two sets of characteristic values of the selected set of coefficients. According to claim 1, And a bit of the payload is inserted by performing one association between one of the feature values of one of the selected set of coefficients and a predefined value. According to claim 1, And a plurality of bits of the payload are inserted by performing a plurality of associations between or among at least two sets of characteristic values of the selected coefficient set. According to claim 1, And a plurality of bits of the payload are inserted by performing a plurality of associations between or among at least two sets of characteristic values of the selected coefficient set. According to claim 1, The modifying step further comprises establishing a constraint-based association between characteristic values of the selected coefficient set based on the payload and key to be inserted into the video volume, further wherein the selected coefficient set is disjointed. (disjoint) A method for inserting a watermark. The method of claim 9, And said association is inequality between said values of said property of said selected disjoint coefficient set. The method of claim 10, Selecting a value r; And Adding the value r to one of the characteristic values of the selected disjoint coefficient set to effect the inequality association Further comprising a watermark. The method of claim 10, And if said inequality association is appropriate, no change of said characteristic value of said selected disjoint coefficient set is performed. The method of claim 10, If an opposite inequality exists and a change in the characteristic value of the selected disjoint coefficient set causes a perceptual damage, no change is made. The method of claim 10, And the characteristic value of the selected disjoint coefficient set is changed to effect the unequal association. The method of claim 10, And the characteristic value of the selected disjoint coefficient set is changed proportionally to effect the unequal association. The method of claim 15, Copying one of said one said characteristic value of said selected disjoint coefficient set; Bit-shifting said one copied characteristic value of said selected disjoint coefficient set; And Adding said one said bit-shifted and copied characteristic value of said selected disjoint coefficient set to said one said characteristic value of said selected disjoint coefficient set Further comprising a watermark. The method of claim 15, Copying said characteristic value of said selected disjoint coefficient set; Bit-shifting the copied characteristic value of the selected disjoint coefficient set; And Adding the bit-shifted and copied feature value of the selected disjoint coefficient set to the corresponding characteristic value of the selected disjoint coefficient set. Further comprising a watermark. The method of claim 9, And the constraint-based association is linear. The method of claim 9, And the constraint-based association is non-linear. The method of claim 9, The constraint-based association is an average value of the characteristic values of the disjoint coefficient set and the characteristic value of the first disjoint coefficient set is changed to a positive value and the value of the characteristic of the second disjoint coefficient set is A method for inserting a watermark, which is changed to negative. The method of claim 9, And the constraint-based association is masked in time. According to claim 1, And the coefficient is changed to a minimum. A system for embedding a watermark by performing an association between feature values of a selected set of coefficients in a video volume, the system comprising: Means for generating a payload; Means for selecting coefficients; Means for changing a coefficient, said coefficient changing means replacing said selected coefficient; And Means for embedding the watermark in the video volume And a watermark for embedding the watermark. The method of claim 23, wherein The characteristic is a measurable characteristic of any portion in the video volume. The method of claim 23, wherein Wherein the characteristic value is calculated by one of averaging, determining a maximum, and determining a minimum. The method of claim 23, wherein The coefficient change is performed in one of a spatial area and a transform area to effect the association between the characteristic values. The method of claim 23, wherein And a bit of the payload is inserted by performing an association between or among at least two sets of feature values of the selected set of coefficients. The method of claim 23, wherein And a bit of the payload is inserted by making one association between one of the characteristic values of one set of the selected set of coefficients and a predefined value. The method of claim 23, wherein And a plurality of bits of the payload are inserted by performing a plurality of associations between or among at least two sets of feature values of the selected coefficient set. The method of claim 23, wherein And a plurality of bits of the payload are inserted by performing a plurality of associations between or among at least two sets of feature values of the selected coefficient set. The method of claim 23, wherein The changing means further comprises establishing a constraint-based association between characteristic values of the selected coefficient set based on the payload and the key to be inserted into the video volume, further wherein the selected coefficient set is disjointed. (disjoint) A system for embedding watermarks. The method of claim 31, wherein And said association is inequality between said values of said property of said selected disjoint coefficient set. 33. The method of claim 32 wherein Means for selecting a value r; And Means for adding said value r to one of said characteristic values of said selected disjoint coefficient set for effecting said unequal association; Further comprising a watermark. 33. The method of claim 32 wherein And if said inequality association is appropriate, no change of said characteristic value of said selected disjoint coefficient set is performed. 33. The method of claim 32 wherein And if there is an opposite inequality association and a change in the characteristic value of the selected disjoint coefficient set causes a perceptual damage, no change is made. 33. The method of claim 32 wherein And the characteristic value of the selected disjoint coefficient set is changed to effect the unequal association. 33. The method of claim 32 wherein And the characteristic value of the selected disjoint coefficient set is changed proportionally to effect the unequal association. The method of claim 37, Means for copying one of said one said characteristic value of said selected disjoint coefficient set; Means for bit-shifting said one copied characteristic value of said selected disjoint coefficient set; And Means for adding said one bit-shifted and copied characteristic value of said selected disjoint coefficient set to said one said characteristic value of said selected disjoint coefficient set The system for inserting a watermark, further comprising. The method of claim 37, Means for copying said characteristic value of said selected disjoint coefficient set; Means for bit-shifting the copied characteristic value of the selected disjoint coefficient set; And Means for adding said bit-shifted and copied characteristic value of said selected disjoint coefficient set to said corresponding characteristic value of said selected disjoint coefficient set. The system for inserting a watermark, further comprising. The method of claim 31, wherein And the constraint-based association is linear. The method of claim 31, wherein And the constraint-based association is non-linear. The method of claim 31, wherein The constraint-based association is an average value of the characteristic values of the disjoint coefficient set and the characteristic value of the first disjoint coefficient set is changed to a positive value and the value of the characteristic of the second disjoint coefficient set is A system for embedding a watermark, which is changed to negative. The method of claim 31, wherein And the constraint-based association is masked in time. The method of claim 23, wherein And the coefficient is changed to a minimum.

Images