DE112020000150T5 - IMPROVED IMAGE WATERMARKS - Google Patents

Abstract

The present disclosure provides systems and methods for improved image watermarking to improve robustness and capacity without degrading perceptibility. In particular, the systems and methods discussed herein enable a higher decoding success rate with the same level of distortion and the same message rate; or a higher message rate with the same level of distortion and the same decoding success rate. Implementations of these systems use a side chain of additional information that is only available to the decoder and not the encoder to achieve asymptotically lossless data compression that enables the same message to be transmitted.

Description

RELATED REGISTRATIONS

The present application claims the advantage and priority of PCT application no. PCT / US2019 / 037959 entitled "Improved Image Watermarking", submitted on June 19, 2019, the entirety of which is hereby expressly incorporated by reference.

BACKGROUND

Image watermarks are a technique that embeds visually imperceptible data or messages in an image and can be categorized as invisible or visible depending on whether the original image is required for watermark extraction. Visible watermark is particularly useful because the embedded data can be recovered without access to the original pre-embedded image.

Current implementations of visible image watermarks, however, can have problems with perceptibility (e.g. whether a distortion introduced by the embedding of the watermark message can be seen by a viewer), robustness (e.g. the success rate with a decoder that decodes the embedded message) and the capacity (e.g., the amount of data or rate at which data can be embedded in the image). In many implementations, increasing one of these implementations can dramatically degrade the other.

SUMMARY

In many implementations, increasing one of these implementations can dramatically degrade the other. In particular, the systems and methods discussed here enable a higher decoding success rate with the same degree of distortion and the same message rate; or a higher message rate with the same degree of distortion and the same decoding success rate. Implementations of these systems use a side chain (or side channel) of additional information that is only available to the decoder and not the encoder to achieve asymptotically lossless data compression, which allows the same message to be transmitted more robustly or in fewer bits.

In one aspect, the present disclosure is directed to a system for enhancing the watermark. The system includes a decoder of a device. The decoder is configured to: receive a capture of an image including at least one embedded watermark; Determining a timestamp of the acquisition; Decoding a binary string from the embedded watermark; Decoding, using part of the timestamp of the acquisition, an identifier from the binary string comprising a timestamp of the image; and outputting the decoded identifier.

In some implementations, the timestamp of the acquisition is identified in metadata of the acquisition. In some implementations, the decoder is configured to extract the timestamp of the detection from a header of a packet that includes the detection. In some implementations, the binary string of the embedded watermark comprises a subset of the timestamp of the image. In another implementation, the decoder is configured to decode the identifier from the binary string by concatenating the portion of the timestamp of the acquisition with the subset of the timestamp of the image. In another further implementation, the binary string of the embedded watermark comprises a number of error correction bits that are greater than a difference between a length of the time stamp of the image and a length of the subset of the time stamp of the image.

In some implementations, the decoder is configured to decode the identifier from the binary string by combining the portion of the timestamp of the acquisition with a predetermined offset. In another implementation, the decoder is configured to decode the identifier from the binary string by iteratively combining the portion of the timestamp of detection with a multiple of the predetermined offset until the identifier is successfully decoded.

In some implementations, the binary string includes an address of a content server that generated the image that includes the at least one embedded watermark. In another Implementation, the binary string includes an identifier of a process of the content server that generated the image that includes the at least one embedded watermark.

In another aspect, the present disclosure is directed to a method of enhancing the watermark. The method includes receiving from a client device a capture of an image including at least one embedded watermark by a decoder of a device. The method also includes determining a timestamp of the detection by the decoder. The method also includes decoding a binary string from the embedded watermark by the decoder. The method also comprises using part of the time stamp of the acquisition, the decoding by the decoder of an identifier from the binary character sequence which comprises a time stamp of the image. The method also includes outputting the decoded identifier by the decoder.

In some implementations, the timestamp of the acquisition is identified in metadata of the acquisition. In some implementations, the method includes extracting, by the decoder, the timestamp of the detection from a header of a packet that includes the detection. In some implementations, the binary string of the embedded watermark comprises a subset of the timestamp of the image. In a further implementation, the method comprises concatenating the part of the timestamp of the acquisition with the subset of the timestamp of the image. In another further implementation, the binary string of the embedded watermark comprises a number of error correction bits that are greater than a difference between a length of the time stamp of the image and a length of the subset of the time stamp of the image.

In some implementations, the method includes combining the portion of the timestamp of the acquisition with a predetermined offset. In another implementation, the method comprises iteratively combining the portion of the timestamp of the acquisition with a multiple of the predetermined offset until the identifier is successfully decoded.

In some implementations, the binary string includes an address of a content server that generated the image that includes the at least one embedded watermark. In a further implementation, the binary string includes an identifier of a process of the content server that generated the image that includes the at least one embedded watermark.

In another aspect, the present disclosure is directed to a watermarking system. The system includes an encoder of a device configured to: receive an image and metadata associated with the image; generate a binary string from a subset of the metadata associated with the image; encode a watermark from the binary string; and embed the watermark in the image. A decoder of the device or of a second device provides the metadata assigned to the image from the subset of the metadata assigned to the image encoded in the embedded watermark and from additional metadata associated with the acquisition of a display of the image on a third device, again.

In some implementations, the metadata associated with the image includes a time stamp of the image and the additional metadata includes a time stamp of the acquisition of the display of the image at the third device. In some implementations, the device's encoder is configured to generate the binary string from a predetermined number of least significant bits of the metadata associated with the image.

In another aspect, the present disclosure is directed to a method of watermarking. The method includes receiving, by an encoder of a device, an image and metadata associated with the image. The method also includes generating a binary string by the encoder from a subset of the metadata associated with the image. The method also includes encoding a watermark from the binary string by the encoder. The method also includes embedding the watermark in the image by the encoder. A decoder of the device or a second device recovers the metadata associated with the image from the subset of metadata associated with the image encoded in the embedded watermark and from additional metadata associated with the acquisition of a display of the image on a third device are.

In some implementations, the metadata associated with the image includes a time stamp of the image and the additional metadata includes a time stamp when the image was captured the third device. In some implementations, the method includes generating the binary string from a predetermined number of least significant bits of the metadata associated with the image.

The present disclosure also provides a computer program that includes instructions that, when executed by a computing device, cause the computing device to perform any of the methods disclosed herein. The present disclosure also provides a computer readable medium that includes instructions that, when executed by a computing device, cause the computing device to perform any of the methods disclosed herein.

Optional features of one aspect can be combined with any other aspect.

Figure list

• 1A ist eine Darstellung einer beispielhaften Implementierung eines Bildwasserzeichens;
• 1B ist eine Darstellung eines Datenformats zur Bildwasserzeichenmarkierung gemäß einer Implementierung;
• 1C ist eine Darstellung eines Datenformats zur Bildwasserzeichenmarkierung gemäß einer anderen Implementierung; und
• 2A ist ein Blockdiagramm eines Systems zum Bildwasserzeichen gemäß einer Implementierung;
• 2B ist ein Blockdiagramm eines Systems zum Bildwasserzeichen gemäß einer anderen Implementierung;
• 3 ist ein Blockdiagramm eines Systems zum Bildwasserzeichen gemäß einigen Implementierungen; und
• 4 ist ein Flussdiagramm eines Verfahrens zum Bildwasserzeichen gemäß einigen Implementierungen.

The details of one or more implementations are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the disclosure will become apparent from the description, drawings, and claims, in which:

• 1A Figure 3 is an illustration of an exemplary implementation of an image watermark;
• 1B Figure 3 is an illustration of a data format for image watermarking according to one implementation;
• 1C Figure 13 is an illustration of a data format for image watermarking in accordance with another implementation; and
• 2A Figure 13 is a block diagram of an image watermarking system according to one implementation;
• 2 B Figure 13 is a block diagram of an image watermarking system in accordance with another implementation;
• 3 Figure 3 is a block diagram of an image watermarking system, in accordance with some implementations; and
• 4th Figure 4 is a flow diagram of a method for image watermarking, in accordance with some implementations.

The same reference numbers and designations in the various drawings refer to the same elements.

DETAILED DESCRIPTION

Image watermarks are a technique that embeds visually imperceptible data or messages in an image and can be categorized as invisible or visible depending on whether the original image is required for watermark extraction. Visible watermark is particularly useful because the embedded data can be recovered without access to the original pre-embedded image.

For example, briefly on 1A Referring to FIG. 3, FIG. 3 is an exemplary implementation of an image watermark for an image 100 shown. A small watermark code 102 may comprise an array of pixels of one size and in an image 100 be placed in such a way that they are not visible to the viewer. As shown, the watermark codes 102 duplicated throughout the image to provide resistance to cropping, regional artifacts from compression or other degradation, or other such distortion. Although a watermark code is shown as having only a few pixels for clarity, in many implementations it may include an area of 64 pixels, 128 pixels, or some other such amount. Adjustments to the pixels to define values of the encoding can be relatively imperceptible as opposed to simply black and white pixels. For example, in many implementations, the pixels that make up the encoded area may have colors that match or are similar to the surrounding pixels, but have adjusted alpha (transparency) values. For example, the encoding can change a pixel with an alpha value of 0 to an alpha value of 10, 50, 100, 255, or some other such value. In some implementations, the code can be recognized by identifying pixels with alpha values that differ greatly from the surrounding alpha values. In some implementations, differential encoding with overlay encoding of each bit, with changes to alpha values of pixels within the overlay, can be used to encode a different value.

Any kind of data can be inside the watermark 102 be coded. In 1B is a data format 150 for the image watermark according to one implementation. The data format shown comprises 128 bits with a 64-bit time stamp 152 (e.g. based on an epoch time), an IP address 154 and a process ID 156 . Data in data format 150 can be referred to here as a query ID. Many implementations also include error correction bits (not shown) to improve the decoding of the watermark. For example, the code can be encoded as a QR code with Reed-Solomon error correcting codes contained in the brand.

In such an implementation, the data may be encoded into an image by a content server prior to providing the image to a client device with an IP address of the content server and a process identifier of the process that created the image. Then, when the image is received and rendered by the client device, a surveillance process on the client device can capture a screenshot of the image and deliver the screenshot to the content server or a surveillance server. For example, the surveillance process on the client device may not be able to access the image itself (e.g., the image may be stored in a location in the client device's memory that the surveillance process cannot access), but it can capture a screenshot of the image (e.g. by reading out image data from a frame buffer or by taking the image with a camera). The server can decode the watermark to identify the original creation process and server and the time the image was created or tagged, and can compare the screenshot image with the original image. This can enable a system to automatically detect distortion or damage to the image caused by rendering or coding processes for the image, as well as other aspects of the image. In particular, in implementations where the content server and the surveillance server are different, this may allow the surveillance server to identify a particular content server of a plurality of content servers that delivered the image to the client device. This can be useful for logging, tracking, and analysis, and is much easier than trying to get HTTP logs or similar logs from the client device (which the monitoring server may not have access to).

Watermarking efficiency can be described in terms of noticeability (e.g., whether distortion introduced by embedding the watermark message can be seen by a viewer), sometimes referred to as "D"; Robustness (e.g., the success rate in a decoder that decodes the embedded message), sometimes referred to as "E"; and capacity (e.g. the amount of data or the rate at which data can be embedded in the image), sometimes referred to as "R". In many implementations, it may be desirable to have low visibility, high robustness, and high capacity. However, in many implementations, improving one of these implementations can cause the others to degrade dramatically. For example, if you want to add more data to the message while maintaining robustness, the watermark may need to be enlarged to make it more noticeable. Similarly, when adding data, the tag size can be maintained by removing error correction bits, although this of course makes the tag more difficult to decode and more susceptible to damage.

2A Figure 3 is a block diagram of a system 200 for image watermarking according to one implementation. The system can have an encoder 202 and a decoder 204 that may reside on the same or different computing devices (e.g., a content server and a monitoring server). A picture "S" 206 can from an encoder 202 with a message "X" 208 encoded to create a watermarked "S" image 210 , which includes S + X. The coded or watermarked image S 'can be transmitted via a communication channel 212 , for example to a client device. A corresponding watermarked image (e.g. from a screenshot, as discussed above) can be sent to the decoder 204 to be provided. For example, the client device can send the watermarked image over the communication channel 212 to the decoder 204 send. The communication channel can thus comprise any combination of networks and devices between the encoder and the decoder, which can possibly introduce any kind of additional distortion. For example, the channel can be lossy due to intentional or unintentional attacks or impairments. Examples of unintended impairments are the rotation of the image, the scaling and the format conversion. Examples of deliberate impairments include injecting noise (e.g. adding information) and attempting to remove watermark codes (e.g. subtracting information).

The decoder 204 can recognize and decode the watermark from the watermarked image S 'to restore the original message X 208', converting an if necessary Apply error correction (and potentially capture multiple watermarks in the image and compare decoded messages from each to eliminate errors or distortions in individual watermarks).

Accordingly, the encoder can encode a message such as the timestamp / address / process ID string discussed above, along with any error correction codes, into a marker such as a QR code; and encode at least one copy of the marker by mixing an alpha channel overlay into an image, and the decoder can decode the message by detecting the overlay pattern and the inconsistencies identifying the QR code; decode the original string; and identify the original timestamp / address / process ID.

Such systems are relatively successful, but have high error rates. In an experiment in which embedded markers were decoded from screenshots of encoded images, the decoding success rate was 44.03%.

As mentioned above, with a fixed message rate (e.g. 128 bit) the factors that influence the decoding success rate are a distortion introduced into the image by the encoder (De) and a distortion between the captured screenshot and the watermarked image on the encoder. Output (Dc). In general, the robustness of the image watermarking, measured in the decoding success rate, is controlled by De: for the same De, the higher the decoding success rate can be achieved if De is increased. However, for most purposes, the watermarks in the watermarked image need not be visually noticeable. Such a requirement imposes an upper bound on De. This constraint on De essentially implies an upper bound on the decoding success rate for any given channel. In some extreme cases where the Dc introduced by the channel is large, the decoding success rate can drop to near zero and thus limit the applicability of such watermark implementations.

(D0, E0, R0) denote the distortion, the decoding success rate and the message rate of implementations of the watermarking method discussed above. In typical implementations, the improvement in one of the three quantities inevitably comes at the expense of the performance loss in at least one of the other quantities. For example, to improve E0, one must forego D0 while retaining R0, or reduce R0 while retaining D0. In many applications, however, both D0 and R0 currently have severe constraints: D0 is necessarily capped to avoid negative effects on the user experience, and R0 is necessarily capped to make the watermark message useful, e.g. for tracking purposes . In this context, current implementations of watermarks have little room for improvement of E0.

The systems and methods discussed here provide for an improved image watermark to improve robustness and capacity without compromising perceptibility. In particular, the systems and methods discussed here enable a higher decoding success rate with the same degree of distortion and the same message rate; or a higher message rate with the same degree of distortion and the same decoding success rate. Implementations of these systems use a side chain of additional information that is only available to the decoder and not the encoder to achieve asymptotically lossless data compression, which allows the same message to be transmitted in fewer bits.

Since the distortion limitation is given by the application, the systems discussed here focus on the compromise between the decoding success rate and the message rate. In particular, the system bypasses the above mentioned lower limit of the message rate without affecting the usefulness of the watermark message. As a result, it allows greater flexibility in finding the right tradeoff between robustness and capacity that was not possible in previous implementations. In particular, page information that is only available at the decoder can be used to achieve asymptotically lossless compression.

2 B Figure 3 is a block diagram of a system 200 ' for marking images with watermarks according to such an implementation. How in connection with 2A encodes an encoder 202 a picture 206 with a message 208 to view a watermarked image 210 to generate that over a communication channel 212 a decoder 204 can be provided. However, to the original message 208 ' the decoder uses additional side information "Y" 214 that are not available to the encoder. This avoids any need for separate communication between the encoder and the decoder, which is common in implementations where the content server and the Monitoring servers are not the same device can be particularly beneficial (and may not be controlled by the same entity).

Between 2A and 2 B the main difference is the introduction of side information Y at the decoder in 2 B . From the classical source coding theorem it follows that the minimum rate required for the lossless recovery of the message X at the decoder in 2A is required, is given by the marginal entropy H (X) of X. Correspondingly, it follows from the Slepian-Wolf coding theorem that the minimum rate required for a lossless recovery of the message X at the decoder in 2 B is required, given by the conditional entropy H (XIY) of X, Y. Since H (XIY) ≤ H (X) for any (X, Y), using the side information Y it is possible to use a lower message rate to get the same message X in 2 B than in 2A transferred to. The stronger the correlation between X and Y, the lower the message rate that can be achieved.

Improved robustness

1. 1. Wandeln der Wasserzeichen-Nachricht X in eine Kbit-Binärzeichenfolge um, wobei K durch H(XIY) bestimmt wird.
2. 2. Wandeln des Kbit-Binärstrings in ein QR-Codewort um.
3. 3. Generieren eines Wasserzeichenbilds, das mindestens eine Kopie des QR-Codeworts enthält.
4. 4. Mischen des Wasserzeichenbilds und des Originalbilds, durch Legen des ersteren über des letzteren.

In a first implementation, the system can use side information Y at the decoder to improve robustness. In some such implementations, the encoder is in 2 B a watermark message embedded in an image like this:

1. 1. Convert the watermark message X to a Kbit binary string, where K is determined by H (XIY).
2. 2. Convert the Kbit binary string into a QR code word.
3. 3. Generate a watermark image that contains at least one copy of the QR code word.
4. 4. Mixing the watermark image and the original image by overlaying the former on top of the latter.

• 5. Erkennen und Extrahieren ein QR-Codewort aus dem Screenshot.
• 6. Decodieren einer Kbit-Binärzeichenfolge aus dem extrahierten QR-Codewort.
• 7. Decodieren der Wasserzeichen-Nachricht X aus dem kbit-Binärstring und den Seiteninformationen Y.

Accordingly, the decoder decodes from 2 B the watermark message X from a screenshot of the watermarked image as follows:

• 5. Recognize and extract a QR code word from the screenshot.
• 6. Decode a kbit binary string from the extracted QR code word.
• 7. Decoding of the watermark message X from the kbit binary string and the page information Y.

It should be noted that in many implementations, one or more of the above steps (e.g., steps 6 through 7) can be combined into a single step for better performance.

It should be noted that a QR code word contains patterns for recognition and an error correction code in a 2D layout. In some embodiments, a 1D error correction codeword along with 1D patterns for recognition can be used in place of a QR codeword for better performance / flexibility in creating a watermark image. Examples of 1D error correcting codes include Reed-Solomon codes, Turbo codes, LDPC (low density parity check) codes, and other general linear block codes.

With regard to step 1 in the coding process above, in order to determine K one must know a priori H (XIY), although knowledge of the realization of Y (i.e. the actual side information sequence) is not required. Examples of side information Y for which a priori knowledge of H (XIY) is available are screenshot timestamps, decoding time and any superfluous information about the screenshot (e.g. information about the website, including its IP address and geographical location, publisher information and platform information).

The following description uses screenshot timestamps as an example, but other similar implementations may use IP address information and / or platform information, or combinations thereof.

It is worth remembering that the in 1B above is a 128-bit binary string consisting of a timestamp (64-bit), an IP address (32-bit) and a process ID (32-bit) (without additional error coding bits). In typical applications, the screenshot timestamp Ts correlates strongly with the timestamp Tq in the query ID, so that Tq ≤ Ts; and there is a non-negative integer Δ with a high probability Ts - Tq ≤ Δ.

In view of this, rather than using 64 bits for the timestamp in such implementations, the encoder in 2B K = Use (ceil (64 - log2 (Δ)) + 64) bits as an estimate of H (XIY), where Y is Ts and "ceil" is an upper bound that rounds its argument up to the nearest whole number. Thus, in one embodiment, a binning scheme is used to encode Tq, with each bin containing candidate timestamps at least microseconds apart, and the index of the bin being the suffix of Tq of length ceil (64 log2 (Δ)).

The proposed binning scheme is based on the fact that the most significant bits are the same for two timestamps that are close to each other. For example, the timestamp for 2019-01-01 in epoch time is: 1546300800, and its binary file is:
0b0101 0111 1110 0101 1010 0011 0101 1110 0110 0110 0000 0000 0000

The timestamp for 2018-01-01 is 1514764800 and its binary version is:
0b0101 0110 0001 1010 1011 1010 1001 1101 0010 1000 0000 0000 0000

The upper 18 bits in their 64-bit representations are the same. The closer the two time stamps are, the more the most significant bits are the same. In typical implementations, the image time stamp and the screenshot time stamp can typically be significantly closer to one another, for example within a day, a week or a month, and thus have a larger number of identical bits.

Using the binning scheme described above, in some implementations the system can use K = (ceil (64 - log2 (Δ)) + 64) bits to encode the query ID by adding approximately log2 (Δ) of the most significant bits in Tq is removed. 1C is an illustration of a data format 150 ' for image watermarking according to such an implementation. As shown are the IP address 154 and the process ID 156 the same as when implementing 1B , with the timestamp on part of the least significant bits 158 is reduced, and additional data 160 can be added without reducing the size of the data.

On the decoder side, the time stamp LSB 158 after obtaining the kbit binary sequence from extracting and decoding the QR code from a received screenshot, identify the index of the bin that contains the correct timestamp Tq. To restore Tq, the decoder can combine the first log2 (Δ) bits of Ts and the (64-log2 (Δ)) bit bin index to get T'q. In cases where log2 (Δ) is not an integer, the smallest integer greater than log2 (Δ), ie ceil (log2 (Δ))), is used here instead. Since there is a high probability of Ts - Tq Δ in many implementations, it follows with a high probability that T'q = Tq at the decoder. In the unlikely event that Ts - Tq> Δ, then as long as Ts - Tq ≤ mΔ, Tq must belong to the following list of size m, where m is a positive integer,
{T'q, T'q - Δ, T'q - 2Δ, ..., T'q - (m-1) Δ}.

1. 1. Erhöhung der Korrekturstufe im ausgewählten QR-Code, z.B. durch Einschließen zusätzlicher Paritäts- oder Fehlerkorrekturbits; oder
2. 2. Verwenden eines kleineren Makro-QR-Codes (z.B. Makro 17).

Since (ceil (64 - log2 (Δ)) + 64) <128, these implementations effectively reduce the message rate required to recover the query ID at the decoder. Such a reduction can in turn be used in two ways to improve the success rate of the decryption:

1. 1. Increase the correction level in the selected QR code, for example by including additional parity or error correction bits; or
2. 2. Use a smaller macro QR code (e.g. macro 17).

It should be noted that a 21x21 QR Code can store up to 152 bits of information, as shown in the following table: ECC level Low middle Quartile High Bits 21x21 152 128 104 72 Numeric 21x21 41 34 27 17th Numerical Macro17 35 30th 21 ---

By reducing the number of bits from 128 to K = (ceil (64- log2 (Δ)) + 64) the system can use a higher error correction code (ECC) level or a smaller QR code to improve the decoding success rate ( e.g. a change from medium to quartii).

Improved message rate

The implementations discussed above use the page information available at the decoder to improve the robustness of the watermarking. From another perspective, the system can also use the side information to improve the message rate.

1. 1. Wandeln der 128-Bit-Abfrage-ID zusammen mit den zusätzlichen KBit-Informationen in einen 128-Bit-Binärstring um, wobei K durch H(X)H(XIY) bestimmt wird.
2. 2. Wandeln des 128-Bit-Binärstring in ein QR-Codewort um.
3. 3. Generieren eines Wasserzeichenbildes, das mindestens eine Kopie des QR-Codeworts enthält.
4. 4. Mischen des Wasserzeichenbildes und des Quellbildes, indem das erstere über das letztere gelegt werden.

In such implementations, the encoder of 2 B embed a watermarked message in an image as follows:

1. 1. Convert the 128-bit query ID together with the additional KBit information into a 128-bit binary string, where K is determined by H (X) H (XIY).
2. 2. Convert the 128-bit binary string into a QR code word.
3. 3. Generating a watermark image that contains at least one copy of the QR code word.
4. 4. Blending the watermark image and the source image by overlaying the former over the latter.

• 4. Erkennen und Extrahieren eines QR-Codewortes aus dem Screenshot.
• 5. Decodieren eines 128-Bit-Binärstrings aus dem extrahierten QR-Codewort.
• 6. Decodieren der 128-Bit-Abfrage-ID zusammen mit den KBit-Zusatzinformationen aus dem 128-Bit-Binärstring und den Nebeninformationen Y.

Correspondingly, the decoder of 2 B decode the watermark message X from a screenshot of the image as follows:

• 4. Recognition and extraction of a QR code word from the screenshot.
• 5. Decoding a 128-bit binary string from the extracted QR code word.
• 6. Decoding of the 128-bit query ID together with the additional KBit information from the 128-bit binary string and the side information Y.

Compared to systems that do not implement these methods, these implementations essentially provide additional K-bit messaging capability for free, i.e. with the same decoding success rate and the same level of distortion. These additional K-bits can be used to provide better tracking and / or user experience in terms of usability.

As mentioned above, although discussed primarily in terms of reducing the data size for a timestamp within the watermark data, a similar implementation with binning that applies to IP addresses and / or process identifiers can be used. For example, if typical process IDs are all less than 20 bits long, 12 bits from the process ID 156 MSB must be removed. Similarly, part of the IP address (e.g. the leftmost 8 bits) in the watermark data can be derived from the side information available on the decoder (e.g. IP address used to send the screenshot, IP address of the decoder, etc. ). To further reduce the data size, combinations of these fields can be processed in this way.

3 Figure 4 is a block diagram of an image watermarking system in accordance with some implementations. A client device 300 , which may include a desktop computer, laptop computer, tablet computer, portable computer, smartphone, embedded computer, smart car, or any other type and form of computing device, may be over a network 312 with one or more servers 314 communicate.

In many implementations, a client device can 300 a processor 302 and a storage device 306 include. The storage device 306 may store machine instructions that, when executed by the processor, cause the processor to perform one or more of the operations described herein. The processor 302 may include a microprocessor, ASIC, FPGA, etc., or combinations thereof. In many implementations, a processor can be a multi-core processor or an array of processors. A storage device 306 may include, but is not limited to, electronic, optical, magnetic, or any other storage device capable of providing program instructions to a processor. A storage device may include a floppy disk, CD-ROM, DVD, magnetic disk, memory chip, ROM, RAM, EEPROM, EPROM, flash memory, optical media, or any other suitable storage device from which a processor can read instructions. The instructions can include, but are not limited to, code from any suitable computer programming language, such as, but not limited to, C, C ++, C #, Java, JavaScript, Perl, HTML, XML, Python, and Visual Basic.

A client device 300 can have one or more network interfaces 304 include. A network interface 304 may include any type or form of interface, including Ethernet, including 10 Base T, 100 Base T, or 1000 Base T (“Gigabit”); any of the flavors of 802.11 wireless, such as 802.11a, 802.11b, 802.11g, 802.11 n, or 802.11 ac; Cellular, including CDMA, LTE, 3G, or 4G cellular; Bluetooth or other short range wireless connections; or any combination of these or other interfaces for communicating with a network. In many implementations, a client device can 300 a variety of network interfaces 304 of various types that involve connections to a variety of networks 312 enable. Accordingly, the network can 312 a local area network (LAN), a wide area network (WAN) such as the Internet, a cellular network, a broadband network, a Bluetooth network, an 802.11 (WiFi) network, a satellite network, or any combination of these or other networks and may include one or include several additional devices (e.g. routers, switches, firewalls, hubs, network accelerators, caches, etc.).

A client device can include one or more user interface devices. A user interface device can be any electronic device that communicates data to a user by using sensory information (e.g. a visualization on a display, one or more sounds, tactile feedback, etc.) and electronic signals (e.g. a keyboard, mouse, a Pointing device, touchscreen display, microphone, etc.). The one or more user interface devices can be located within the housing of a client device, such as a built-in display, touch screen, microphone, etc., or outside the housing of a client device, such as a monitor that communicates with a client device. Device, a speaker connected to a client device, etc., according to various implementations.

The memory 306 can be an application 308 for execution by the process 302 include. The application 308 may include any type and form of application such as a media application, web browser, productivity application, or any other such application. The application 308 can receive images from a content server, including watermarks embedded in the images, and display them through a user interface to a user of the client device.

The memory 306 can also be a capture machine 310 which is part of the application 308 (eg a plug-in or an extension of a browser) and / or part of an operating system of the device can include. The capture machine 310 may include an application, server, service, daemon, routine, or other executable logic for capturing screenshots of rendered images including watermarks. The capture machine 310 can be configured to take a screenshot of each image or some images. In some implementations, the acquisition engine can 310 for example triggered to take a screenshot of an image in response to metadata of the image, or in response to an, from the application 308 to record executed script (e.g. in response to a script embedded in a web page that is displayed by a browser, for example). The capture machine 310 can be configured to take a screenshot of each image or some images. In some implementations, the acquisition engine can 310 for example triggered to take a screenshot of an image in response to metadata of the image or in response to one from the application 308 to record executed script (e.g. in response to a script embedded in a web page that is displayed by a browser, for example). The capture machine 310 can add metadata to the screenshot, such as a recording time (e.g. in epoch time), as discussed above. The capture machine 310 can also take the screenshot via the network interface 304 transmitted to a monitoring server. In some implementations, the acquisition engine can 310 comprise a script embedded in a web page and from an application 308 runs while the webpage is being rendered; such web pages may also contain an embedded image or a link to an image that the capture engine can take a screenshot of.

The server 314 may include a content server and / or a monitoring server, which may be the same or different devices. The servers 314 can be one or more processors 302 , Network interfaces 304 and storage devices 306 include. A content server 314 can be one or more content items 316 in memory such as images to be watermarked and other content (e.g. web pages, other media, etc.). The content server 314 can also use an encoder 202 include, as above in connection with the 2A and 2 B was discussed. The encoder 202 may include software, hardware, or a combination of hardware and software. For example, the encoder can 202 include an ASIC, FPGA, or other dedicated hardware for embedding watermarks in images.

A monitoring server can have a decoder 204 include, as above in connection with 2 B discussed. The decoder 204 may include software, hardware, or a combination of hardware and software. For example, the decoder can 204 include an ASIC, FPGA, or other dedicated hardware for identifying and decoding watermarks from images. As discussed above, the decoder can 204 Received side information to aid in decoding the watermark, such as a screenshot time from metadata of a screenshot taken by a capture engine 310 was received.

4th Figure 4 is a flow diagram of a method for image watermarking, in accordance with some implementations. At step 402 a client device can request an item of content. The request can be triggered by a browser or other application while a web page is being rendered (e.g. an intermediate content item during a break in a mobile game or any other type and form of content). At step 404 can be a content server 314 select a content item. The content item can be selected in any way and can be based on the client device type, a user account or device identifier, contextual elements within a web page or other application, or other such information.

In step 406 can the content server 314 generate a watermark identifier, which may include one or more identifiers, including a time stamp, an identifier of the server or an IP address of the server, and / or a process identifier of the process used to select the content item. In some implementations, the watermark identifier may contain additional information, such as an identifier of the content item. At step 408 the content element can be coded with the watermark. As discussed above, encoding the content item may include creating an overlay with an alpha channel having pixels modified from a default value or pattern representing changed bits of the encoded watermark (e.g., a QR code or similar code). The watermarks can be repeated across the image at predetermined intervals. The overlay can then be blended or combined with the image to create the encoded content item. At step 410 the encoded content item can be transmitted from the content server to the client device.

Although the content server has been shown to generate the watermark identifier and encode the watermark for the content item after receiving the request, in some implementations content items may be precoded (e.g., before step 402 ) and the content server can select a precoded content item for delivery. In many implementations, such precoding can be performed within a predetermined time frame prior to the request. For example, content items can be encoded with a given time stamp and used for a predetermined period of time (eg two weeks) and then replaced with a new time stamp or re-encoded. This can allow the content server to perform the coding processing during less busy times while still ensuring that the content and timestamp are relatively fresh. As discussed above, the shorter the window during which precoded content elements can be used, the more data can be encoded in the watermark and / or the watermark can be made more robust; however, even in the example described above, windows of a year or more can be used while still significantly reducing the data required.

In step 412 the client device can render the content item, for example within an application such as a web browser, media player, game or other application. At step 414 For example, a capture engine on the client device (running either as a separate service or as a plug-in or script of the application) can capture a screenshot of the content item. The screenshot can be cropped or limited to the content element, or it can include all or part of the screen. The screenshot can be identified using metadata with a recording time stamp and can contain other identifiers (e.g. device identifiers, context identifiers of the application and / or website, etc.). The screenshot can be cropped or limited to the content element, or it can include all or part of the screen. The screenshot can be identified using metadata with a recording time stamp and can contain other identifiers (e.g. device identifiers, context identifiers of the application and / or website, etc.). At step 416 the client device can transmit the screenshot to a monitoring server.

At step 418 For example, a monitoring server, which can be a content server or other device, can receive the screenshot and, in some implementations, can extract the timestamp from the metadata of the screenshot or identify a time the screenshot was sent or received. The time stamp can be supplied as side information to a decoder of the monitoring server.

In step 420 the decoder can scan the screenshot and extract all identified watermarks. In some implementations where the watermark appears multiple times in the screenshot, the decoder can compare the identified watermarks and select or generate a watermark with the least distortion (e.g. a watermark that corresponds to the highest number of other watermarks in the image, a watermark that has a Average of other identified watermarks, etc.). At step 422 the decoder can convert the watermark into a character string.

In step 424 the decoder can create a timestamp from part of the extracted timestamp from step 418 generate (e.g. a predetermined number of the least significant bits) and can test the decoding of the character string using the generated time stamp (e.g. applying error correction algorithms to the decoded character string with the generated time stamp). If the character string is correctly decoded according to the error correction bits, the monitoring server can go to step 426 Process the screenshot image or data relating to the content element (e.g. identify the content server via the IP address and process identifier, compare the screenshot with the original content element in order to detect rendering distortions or damage, tracking the delivery of the content element, etc. ).

If the character string is not correctly decoded, the decoder can advance the generated time stamp according to the value Δ from the binning scheme and can decode at step 424 test again. This can be repeated iteratively until either the decoding is successful or all bin index values have been tested (suggesting that the watermark has been damaged or improperly extracted, or that an item of content from the usage window discussed above has been used prior to use). If all bin index values have been tested and the decoding is unsuccessful, then the decoder in step 428 report an error to an administrator or user of the system.

Accordingly, the systems and methods discussed herein provide improved image watermarking to improve robustness and capacity without degrading perceptibility. In particular, the systems and methods discussed herein enable a higher decoding success rate with the same level of distortion and the same message rate; or a higher message rate with the same level of distortion and the same decoding success rate. Implementations of these systems use a side chain of additional information that is only available to the decoder and not the encoder to achieve asymptotically lossless data compression, which allows the same message to be transmitted in fewer bits.

Implementations of the subject matter and the operations described in this specification may be implemented in digital electronic circuits or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of: or more thereof . Implementations of the subject matter described in this specification can be implemented as one or more computer programs, i.e. one or more modules of computer program instructions stored on one or more computer storage media for execution by or to control the operation of data processing equipment. Alternatively or additionally, the program instructions can be encoded on an artificially generated propagated signal, e.g. A computer storage medium can be or be included in a computer readable storage device, a computer readable storage substrate, a storage array or device with random or serial access, or a combination of one or more thereof. While a computer storage medium is not a broadcast signal, a computer storage medium can also be a source or destination of computer program instructions encoded in an artificially generated broadcast signal. The computer storage medium can also be or be included in one or more separate components or media (e.g., multiple CDs, disks, or other storage devices). Accordingly, the computer storage medium can be tangible.

The operations described in this specification can be implemented as operations performed by a computing device on data stored on one or more computer-readable storage devices or received from other sources.

The term “client” or “server” encompasses all types of devices, devices and machines for processing data, such as a programmable processor, a computer, a system on a chip, or several or combinations of the foregoing. The device can be special Logic circuits, for example an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, the device may also contain code that generates an execution environment for the computer program in question, e.g. code that forms processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine or a combination of one or more thereof . The device and execution environment can implement various different computer model infrastructures, such as web services, distributed computing and grid computing infrastructures.

A computer program (also known as a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and it can be written in any form, including as a stand-alone program or as a module , Component, subroutine, object, or other entity suitable for use in a computing environment. A computer program can, but does not have to, correspond to a file in a file system. A program can be stored in part of a file that contains other programs or data (e.g. one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files ( e.g. files that store one or more modules, subprograms or parts of code). A computer program can be used to run on one computer or on several computers that are located at one location or that are distributed over several locations and are connected to one another by a communication network.

The processes and logic described in this specification can be executed by one or more programmable processors executing one or more computer programs to perform actions by manipulating input data and generating output. The processes and logic operations can also be carried out by special logic circuits, e.g. an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit), and the device can also be implemented as a special logic circuit.

Processors suitable for executing a computer program include both general purpose and specialty microprocessors and one or more processors of any type of digital computer. Generally, a processor receives instructions and data from read-only memory or random access memory, or both. The essential elements of a computer are a processor for executing actions according to instructions and one or more storage devices for storing instructions and data. In general, a computer also includes one or more mass storage devices for storing data, such as magnetic, magneto-optical disks, or optical disks, or is operatively coupled to receive or transmit data from, or to, or both. However, a computer does not need to have such devices. In addition, a computer can be embedded in another device, such as a cell phone, Personal Digital Assistant (PDA), portable audio or video player, game console, Global Positioning System (GPS) receiver, or portable storage device (such as a Universal Serial Bus USB Flash Drive, to name a few. Devices suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and storage devices, including semiconductor storage devices such as EPROM, EEPROM, and flash memory devices; Magnetic disks, e.g. internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by or built into special logic circuits.

To enable interaction with a user, you can Implementations of the subject matter described in this specification can be implemented on a computer with a display device, e.g. a CRT (cathode ray tube), LCD (liquid crystal display), OLED (organic light emitting diode), TFT (thin film transistor), plasma, another flexible configuration or any other monitor for Displaying information to the user and a keyboard, pointing device such as a mouse, trackball, etc., or a touch screen, touchpad, etc. through which the user can input into the computer. Other types of devices can also be used to provide interaction with a user; Feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including audible, spoken, or tactile input. In addition, a computer can interact with a user by sending and receiving documents to a device that is used by the user; by sending web pages to a web browser on a user's client device in response to requests received from the web browser.

Implementations of the subject matter described in this specification can be implemented in a computer system that includes a back-end component, for example as a data server, or that includes a middleware component, for example an application server, or that includes: a front-end component , e.g. a client computer with a graphical user interface or a web browser through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back-end, middleware or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Communication networks can include a local area network ("LAN") and a wide area network ("WAN"), an internetwork (e.g., the Internet), and peer-to-peer networks (e.g., ad-hoc peer-to-peer networks).

For situations in which the systems discussed herein can collect personal information about users or use personal information, users can be given the option to control whether programs or functions that collect personal information (e.g. information about the user's social network, social actions) or activities, preferences of a user, or a user's location) or to control whether or how content is received from a content server or other computing system that might be more relevant to the user. In addition, certain data can be anonymized in one or more ways before being stored or used, so that personal data is removed when the parameters are generated. For example, the identity of a user can be anonymized so that no personal data can be determined for the user, or the geographic location of a user can be generalized when location information is obtained (e.g. at the city, zip code or state level) so that a specific Location of a user cannot be determined. Thus, the user can have control over how information about him is collected and used by the content server.

While this description contains many specific implementation details, these should not be construed as limitations on the scope of inventions or what can be claimed, but rather as descriptions of features that are specific to particular implementations of certain inventions. Certain features that are described in the context of separate implementations in this specification can also be implemented in combination in a single implementation. Conversely, various features that are described in the context of a single implementation can also be implemented in multiple implementations separately or in any suitable sub-combination. In addition, although features can be described above as acting in certain combinations and even originally claimed as such, in some cases one or more features from a claimed combination can be removed from the combination, and the claimed combination can refer to a sub-combination or variation of a Subcombination be directed.

Similarly, while operations are depicted in the drawings in a particular order, it should not be understood that such operations must be performed in the particular order shown, or in sequential order, or all of the operations illustrated, to achieve the desired result. Multitasking and parallel processing can be beneficial. Furthermore, the separation of various system components in the implementations described above should not be understood as requiring such separation in all implementations, and it should be understood that the described program components and systems are generally integrated together in a single software product or in a package multiple software products can be packaged.

Thus, certain implementations of the subject matter have been described. Other implementations are within the scope of the following claims. In some cases, the measures recited in the claims can be carried out in a different order and still achieve desirable results. In addition, the processes illustrated in the accompanying figures do not necessarily require the particular order or sequential order shown in order to achieve the desired results. In certain implementations, multitasking or parallel processing can be used.

QUOTES INCLUDED IN THE DESCRIPTION

This list of the documents listed by the applicant was generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Patent literature cited

• US 2019/037959 [0001]

Claims (27)

A decoder for a watermarking system, the decoder configured to: receive from a client device a packet comprising (i) a screenshot of an image displayed by the client device, the image having a watermark associated with metadata associated with the image and (ii) metadata associated with the screenshot of the image; extract from the package the metadata associated with the screenshot of the image and the string; decode an identifier from the character string using a portion of the metadata associated with the screenshot of the image, the identifier comprising metadata associated with the image; and track the delivery of an item of content associated with the image in response to the decoded identifier. System according to Claim 1 wherein the metadata associated with the screenshot of the image comprises a time stamp of the screenshot of the image, and wherein the decoded identifier comprises a time stamp of the image. System according to Claim 2 wherein the decoder is configured to extract the timestamp of the screenshot of the image from a header of the packet that includes the screenshot of the image and the metadata associated with the screenshot of the image. System according to Claim 2 or 3 , wherein the string comprises a subset of the timestamp of the image. System according to Claim 4 wherein the decoder is configured to decode the identifier from the string by concatenating a portion of the timestamp of the screenshot of the image with the subset of the timestamp of the image. System according to Claim 4 or Claim 5 wherein the character string comprises a number of error correction bits greater than a difference between a length of the time stamp of the image and a length of the subset of the time stamp of the image. The system of any preceding claim, wherein the decoder is configured to decode the identifier from the character string by combining the portion of the metadata associated with the screenshot of the image with a predetermined offset. System according to Claim 7 wherein the decoder is configured to decode the identifier from the character string by iteratively combining the portion of the metadata associated with the screenshot of the image with a multiple of the predetermined offset until the identifier is successfully decoded. The system of any preceding claim, wherein the character string comprises an address of a content server that generated the watermarked image with the character string. System according to Claim 9 wherein the character string comprises an identifier of a process of the content server that generated the watermarked image with the character string. A method of applying watermarks comprising: Receiving a packet by a decoder of a device from a client device, the packet comprising (i) a screenshot of an image displayed by the client device, watermarking the image having metadata associated with the image is linked, is linked and (ii) metadata linked to the image is the screenshot of the image; Extracting, by the decoder, from the packet the metadata associated with the screenshot of the image and the string; Using a portion of the metadata associated with the screenshot of the image, the decoder decoding an identifier from the character string, the identifier comprising metadata associated with the image; and Tracking, by the decoder, the delivery of an item of content associated with the image in response to the decoded identifier. Procedure according to Claim 11 wherein the metadata associated with the screenshot of the image comprises a time stamp of the screenshot of the image, and wherein the decoded identifier comprises a time stamp of the image. Procedure according to Claim 12 , further comprising extracting, by the decoder, the timestamp of the screenshot of the image from a header of the packet comprising the screenshot of the image. Method according to 12 or 13, wherein the character string comprises a subset of the time stamp of the image. Procedure according to Claim 14 wherein decoding the identifier from the character string further comprises concatenating a portion of the timestamp of the screenshot of the image with the subset of the timestamp of the image. Procedure according to Claim 14 or Claim 15 wherein the character string comprises a number of error correction bits greater than a difference between a length of the time stamp of the image and a length of the subset of the time stamp of the image. Method according to one of the Claims 11 until 16 wherein decoding the identifier from the string further comprises combining the portion of the metadata associated with the screenshot of the image with a predetermined offset. Procedure according to Claim 17 wherein decoding the identifier from the string further comprises iteratively combining the portion of the metadata associated with the screenshot of the image with a multiple of the predetermined offset until the identifier is successfully decoded. Method according to one of the Claims 11 until 18th wherein the character string comprises an address of a content server that generated the watermarked image with the character string. Procedure according to Claim 19 wherein the character string comprises an identifier of a process of the content server that generated the watermarked image with the character string. A computer readable medium comprising instructions that, when executed by a computing device, cause the computing device to use the method of any of the Claims 11 until 20th perform. Watermarking system comprising: an encoder of a device configured to: receive an image and metadata associated with the image, generate a string from a subset of the metadata associated with the image, encode a watermark from the string and embed the watermark in the image; being a decoder of the device or a second device receives a packet from a client device comprising (i) a screenshot of the image displayed by the client device, the image is watermarked with the character string generated from the subset of the metadata associated with the image, and ( ii) metadata of the image associated with the screenshot; from the package that extracts the screenshot of the image displayed by the client device and includes the metadata associated with the screenshot of the image, the string and additional metadata associated with the screenshot of the image; using part of the additional metadata assigned to the screenshot of the image, decoding an identifier from the character string, the identifier comprising a portion of the metadata assigned to the image; and tracks the delivery of an item of content associated with the image in response to the decoded identifier. System according to Claim 22 wherein the metadata associated with the image comprises a time stamp of the image, and wherein the additional metadata comprises a time stamp of the screenshot of the image by the client device. System according to Claim 22 or 23 wherein the encoder of the device is configured to generate the character string from a predetermined number of least significant bits of the metadata associated with the image. A method of applying watermarks, comprising: receiving, by an encoder of a device, an image and metadata associated with the image; Generating, by the encoder, a character string from a subset of the metadata associated with the image; Encoding, by the encoder, a watermark from the character string; and embedding, by the encoder, the watermark in the image; wherein a decoder of the device or a second device receives from a client device a packet comprising (i) a screenshot of the image displayed by the client device, the image that is watermarked with the character string selected from the subset of the Metadata associated with the image has been generated, is provided, and (ii) metadata of the image associated with the screenshot; from the package that extracts the screen shot of the image displayed by the client device and includes the metadata associated with the screen shot of the image, the character string and additional metadata associated with the screen shot of the image; using part of the additional metadata assigned to the screenshot of the image, decoding an identifier from the character string, the identifier comprising a portion of the metadata assigned to the image; and track delivery of an item of content associated with the image in response to the decoded identifier. Procedure according to Claim 25 wherein the metadata associated with the image comprises a time stamp of the image, and wherein the additional metadata comprises a time stamp of the screenshot of the image by the client device. Procedure according to Claim 25 or 26th further comprising generating the character string from a predetermined number of least significant bits of the metadata associated with the image.

Images