JP7225403B2 - Improved image watermarking - Google Patents

Description

The present application relates to systems and methods for improved image watermarking.
[Related Application]
This application claims the benefit of and priority to International Application No. PCT/US2019/037959, filed June 19, 2019, entitled "Improved Image Watermarking", the entirety of which is incorporated herein by reference. incorporated into.

Image watermarking is a technique of embedding visually imperceptible data or messages into an image and can be classified as non-blind or blind, depending on whether the original image is required for watermark extraction, respectively. Blind watermarking is particularly useful in that embedded data can be recovered without access to the original pre-embedded image.

However, current implementations of blind image watermarking are subject to perceptibility (e.g., whether the distortion introduced by embedding the watermarking message can be detectable by a viewer), robustness (e.g., embedded success rate at the decoder to decode the message), and capacity (eg, the amount of data or the rate at which data can be embedded in an image). In many implementations, increasing one of these can lead to dramatic degradation of the others.

The systems and methods described herein provide improved image watermarking for improved robustness and capacity without reducing perceptibility. Specifically, the systems and methods described herein enable higher decoding success rates at the same distortion level and message rate, or higher message rates at the same distortion level and decoding success rate. enable. Implementations of these systems take advantage of additional information sidechains (or sidechannels) available only to the decoder, not the encoder, to achieve asymptotically lossless data compression, making the same message more robust. , or fewer bits.

In one aspect, the present disclosure is directed to a system for improved watermarking. The system includes the decoder of the device. A decoder receives a capture of an image containing at least one embedded watermark, determines the timestamp of the capture, decodes the binary string from the embedded watermark, and uses part of the timestamp of the capture. to decode the identifier from the binary string containing the timestamp of the image and output the decoded identifier.

In some implementations, the timestamp of the capture is identified within the metadata of the capture. In some implementations, the decoder is configured to extract the capture timestamp from the header of the packet containing the capture. In some implementations, the embedded watermark binary string includes a subset of the image timestamps. In a further implementation, the decoder is configured to decode the identifier from the binary string by concatenating a portion of the timestamp of the capture with a subset of the timestamps of the image. In another further implementation, the embedded watermark binary string includes a number of error correction bits greater than the difference between the length of the image timestamp and the length of a subset of the image timestamps. .

In some implementations, the decoder is configured to decode the identifier from the binary string by combining a portion of the timestamp of the capture with a predetermined offset. In a further implementation, the decoder is configured to decode the identifier from the binary string by iteratively combining portions of the timestamp of the capture with multiples of the predetermined offset until the identifier is successfully decoded. be.

In some implementations, the binary string includes the address of the content server that generated the image with at least one embedded watermark. In a further implementation, the binary string includes an identifier of the content server process that generated the image including at least one embedded watermark.

In another aspect, the present disclosure is directed to methods for improved watermarking. The method includes receiving, by a decoder of the device, a capture of an image including at least one embedded watermark from a client device. The method also includes determining, by the decoder, the timestamp of the capture. The method also includes decoding, by a decoder, the binary string from the embedded watermark. The method also includes using a portion of the timestamp of the capture to decode, by a decoder, the identifier from the binary string containing the timestamp of the image. The method also includes outputting, by a decoder, the decoded identifier.

In some implementations, the timestamp of the capture is identified within the metadata of the capture. In some implementations, the method includes extracting, by a decoder, a capture timestamp from a header of a packet containing the capture. In some implementations, the embedded watermark binary string includes a subset of the image timestamps. In a further implementation, the method includes concatenating a portion of the capture timestamps with a subset of the image timestamps. In another further implementation, the embedded watermark binary string includes a number of error correction bits greater than the difference between the length of the image timestamp and the length of a subset of the image timestamps. .

In some implementations, the method includes combining a portion of the capture timestamp with a predetermined offset. In a further implementation, the method includes iteratively combining a portion of the timestamp of the capture with multiples of the predetermined offset until the identifier is successfully decoded.

In some implementations, the binary string includes the address of the content server that generated the image with at least one embedded watermark. In a further implementation, the binary string includes an identifier of the content server process that generated the image including at least one embedded watermark.

In another aspect, the disclosure is directed to a watermarking system. The system receives an image and metadata associated with the image, generates a binary string from a subset of the metadata associated with the image, and encodes the watermark from the binary string. and embedding the watermark into the image. A decoder on the device or the second device may extract a subset of the metadata associated with the image encoded in the embedded watermark and additional metadata associated with capturing the display of the image on the third device. , recover the metadata associated with the image.

In some implementations, the metadata associated with the image includes a timestamp of the image and the additional metadata includes a timestamp of capture of the display of the image on the third device. In some implementations, the device's encoder is configured to generate a binary string from a predetermined number of least significant bits of metadata associated with the image.

In another aspect, the disclosure is directed to a method for watermarking. The method includes receiving, by an encoder of the device, an image and metadata associated with the image. The method also includes generating, by an encoder, a binary string from the subset of metadata associated with the image. The method also includes encoding the watermark from the binary string by an encoder. The method also includes embedding the watermark in the image by the encoder. A decoder on the device or the second device may extract a subset of the metadata associated with the image encoded in the embedded watermark and additional metadata associated with capturing the display of the image on the third device. , recover the metadata associated with the image.

In some implementations, the metadata associated with the image includes a timestamp of the image and the additional metadata includes a timestamp of capture of the display of the image on the third device. In some implementations, the method includes generating a binary string from a predetermined number of least significant bits of metadata associated with the image.

The present disclosure also provides a computer program product containing instructions that, when executed by a computing device, causes the computing device to perform any of the methods disclosed herein. The present disclosure also provides a computer-readable medium containing instructions that, when executed by a computing device, cause the computing device to perform any of the methods disclosed herein.

Optional features of one aspect may be combined with any other aspect.

The details of one or more implementations are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the disclosure will become apparent from the description, drawings, and claims.

FIG. 4 is a diagram of an exemplary implementation of image watermarking; FIG. 4 is a diagram of a data format for image watermarking, according to one implementation; FIG. 4 is a diagram of a data format for image watermarking, according to another implementation; 1 is a block diagram of a system for image watermarking, according to one implementation; FIG. FIG. 4 is a block diagram of a system for image watermarking, according to another implementation; 1 is a block diagram of a system for image watermarking, according to some implementations; FIG. 4 is a flowchart of a method for image watermarking, according to some implementations;

Like reference numbers and designations in the various drawings indicate like elements.

Image watermarking is a technique of embedding visually imperceptible data or messages into an image and can be classified as non-blind or blind, depending on whether the original image is required for watermark extraction, respectively. Blind watermarking is particularly useful in that embedded data can be recovered without access to the original pre-embedded image.

For example, briefly referring to FIG. 1A, an exemplary implementation of image watermarking for image 100 is shown. The small watermark code 102 can comprise an array of pixels of a certain size and is placed within the image 100 so as to be invisible to the viewer. As shown, the watermark code 102 may be replicated throughout the image to provide immunity to local artifacts due to cropping, compression or other impairments, or other such distortions. Although shown with only a few pixels for clarity, in many implementations the watermark code may include areas having 64 pixels, 128 pixels, or any other such amount. Adjustments to pixels to define encoding values can be relatively imperceptible as opposed to simple black and white pixels. For example, in many implementations, the pixels that make up the encoded region have colors that match or are similar to surrounding pixels, but may have adjusted alpha (transparency) values. For example, encoding may change a pixel with an alpha value of 0 to an alpha value of 10, 50, 100, 255, or any other such value. In some implementations, the code may be detected by identifying pixels with alpha values that differ significantly from surrounding alpha values. In some implementations, differential encoding can be applied to overlay encode each bit by changing the alpha value of the pixels in the overlay to encode different values.

Any kind of data can be encoded within the watermark 102 . Referring briefly to FIG. 1B, a data format 150 for image watermarking is shown, according to one implementation. The illustrated data format includes 128 bits, with a 64-bit timestamp 152 (eg, based on epoch time), IP address 154, and process identifier 156. FIG. The data in data format 150 is sometimes referred to herein as a query ID. Many implementations also include error correction bits (not shown) to improve decoding of the watermark. For example, the code may be encoded as a QR Code™ with a Reed-Solomon error correction code contained within the mark.

In one such implementation, the data is encoded into the image by the content server using the IP address of the content server and the process identifier of the process that generated the image prior to serving the image to the client device. good too. Thereafter, when the image is received and rendered by the client device, a monitoring process on the client device may capture a screenshot of the image and provide the screenshot to the content server or monitoring server. For example, a monitoring process on a client device may not be able to access the image itself (e.g., the image may be stored in a location in the client device's memory that the monitoring process cannot access), but (e.g., a frame You can capture a screenshot of the image (either by reading the image data from a buffer or by capturing the image with the camera). The server can decode the watermark to identify the original generating process and server, as well as the time the image was generated or marked, and can compare the screenshotted image with the original image. This allows the system to automatically identify distortions or image corruption caused by the image rendering or encoding process, and to identify other aspects of the image. In implementations where the content server and monitoring server are different, this may allow, among other things, the monitoring server to identify the particular content server among multiple content servers that provided the image to the client device. This can be useful for logging, tracking, and analysis, and can be considerably easier than trying to retrieve HTTP logs or similar logs from client devices (which the monitoring server may not be able to access).

Watermarking efficiency is measured by perceptibility, sometimes referred to as 'D' (e.g., whether the distortion introduced by embedding the watermarking message can be detectable by a viewer), and robustness, sometimes referred to as 'E'. for example, the rate of success at a decoder decoding the embedded message), and capacity, sometimes referred to as 'R' (eg, the amount of data or the rate at which data can be embedded in an image). In many implementations, it may be desirable to have low perceptibility, high robustness, and high capacity. However, in many implementations, improving one of these can lead to dramatic degradation in the other. For example, adding more data to a message while maintaining robustness may require increasing the size of the watermark to make it more perceptible. Similarly, the mark size can be maintained while adding data by removing error correction bits, but this of course makes the mark more difficult to decode and more prone to corruption.

FIG. 2A is a block diagram of a system 200 for image watermarking, according to one implementation. The system may include encoder 202 and decoder 204, which may be on the same or different computing devices (eg, content server and monitoring server). Image 'S' 206 can be encoded with message 'X' 208 by encoder 202 to create watermarked image 'S'' 210 containing S+X. Encoded or watermarked image S′ may be transmitted over communication channel 212, such as to a client device. A corresponding watermarked image (eg, from a screenshot as described above) may be provided to decoder 204 . For example, a client device may send a watermarked image to decoder 204 via communication channel 212 . Accordingly, the communication channel can include any combination of networks and devices between the encoder and decoder, which can potentially introduce any kind of additional distortion. For example, channels may suffer losses as a result of intentional or unintentional attacks or failures. Examples of unintended failures are image rotation, scaling, and format conversion. Examples of intentional interference include attempts to inject noise (eg add information) and remove watermark codes (eg subtract information).

The decoder 204 detects the watermark from the watermarked image S', decodes it to recover the original message X 208', and applies error correction if necessary (and potentially multiple watermarks in the image). , and compare the decoded messages from each to rule out errors or distortions in a single watermark).

Therefore, the encoder encodes a message such as the timestamp/address/process ID string mentioned above, along with any error correction code, into a mark, such as a QR code, and then encodes at least one of the marks via blending of alpha channel overlays. Encoding the copy into an image, the decoder decodes the message by detecting inconsistencies and overlay patterns that identify the QR code, decodes the original string, and identifies the original timestamp/address/process ID. obtain.

Such systems are relatively successful, but have high error rates. In one experiment involving decoding embedded marks from screenshots of encoded images, the decoding success rate was 44.03%.

As mentioned above, given a fixed message rate (e.g., 128 bits), the factors affecting the decoding success rate are the distortion introduced into the image by the encoder (D _e ), and the encoder output (D _c ) between the captured screenshot and the watermarked image. In general, image watermarking robustness, measured by decoding success rate, is controlled by D _e , and for the same D _c , higher decoding success rate can be achieved as D _e increases. However, for most purposes the watermark should be visually imperceptible in the watermarked image. Such a requirement imposes an upper bound on D _e . This constraint on D _e essentially implies an upper bound on the decoding success rate for any given channel. In some extreme cases where _Dc introduced by the channel is large, the decoding success rate may drop to nearly zero, thus limiting the applicability of such implementations of watermarking.

Let (D ₀ , E ₀ , R ₀ ) denote the distortion, decoding success rate, and message rate of the implementation of the watermarking method described above, respectively. In a typical implementation, improvement in one of the three quantities will necessarily come at the expense of performance loss in at least one of the other quantities. For example, to improve _E0 , one needs to sacrifice _D0 while maintaining _R0 , or reduce _R0 while maintaining _D0 . However, in many applications both _D0 and _R0 are currently severely constrained, with _D0 necessarily having an upper bound to avoid adversely affecting the user experience, and _R0 being e.g. For the purpose watermarking messages are useful, they necessarily have a lower bound. In this context, current implementations of watermarking have little room to improve _E0 .

The systems and methods described herein provide improved image watermarking for improved robustness and capacity without reducing perceptibility. Specifically, the systems and methods described herein enable higher decoding success rates at the same distortion level and message rate, or higher message rates at the same distortion level and decoding success rate. enable. Implementations of these systems take advantage of sidechains of additional information available only to the decoder, not the encoder, to achieve asymptotically lossless data compression, so that the same message is sent with fewer bits. enable

Since the distortion constraint is given by the application, the system described herein focuses on the trade-off between decoding success rate and message rate. Specifically, the system avoids the aforementioned lower bounds on message rates without compromising the usefulness of watermarking messages. As a result, it allows greater flexibility in finding the right trade-off between robustness and capacity not possible with conventional implementations. Specifically, side information available only at the decoder can be used to asymptotically achieve lossless compression.

FIG. 2B is a block diagram of a system 200' for image watermarking, according to one such implementation. As described in connection with FIG. 2A, encoder 202 encodes image 206 with message 208 to produce watermarked image 210 that can be provided to decoder 204 via communication channel 212 . However, to recover the original message 208', the decoder uses additional side information 'Y' 214 that is not available to the encoder. This eliminates the need for separate communications between the encoder and decoder, especially in implementations where the content server and monitoring server are not the same device (and may not be controlled by the same entity). can be advantageous.

The main difference between FIG. 2A and FIG. 2B is the introduction of side information Y in the decoder of FIG. 2B. From the classical source coding theorem, it can be seen that the minimum rate required for lossless recovery of message X in the decoder of FIG. 2A is given by the marginal entropy of X, H(X). Correspondingly, from the Slepian-Wolf coding theorem, the minimum rate required for lossless recovery of message X in the decoder of FIG. 2B is given by the conditional entropy H(X|Y) of X given Y I understand. Since H(X|Y)≤H(X) for any (X, Y), by using side information Y, FIG. It is possible to communicate. The stronger the correlation between X and Y, the lower the message rate can be.

Improving Robustness In a first implementation, the system may exploit side information Y at the decoder to improve robustness. In some such implementations, the encoder of FIG. 2B embeds the watermarking message into the image as follows.
1. Convert the watermarking message X to a K-bit binary string, where K is determined by H(X|Y).
2. Convert K-bit binary string to QR codeword.
3. Generate a watermarking image containing at least one copy of the QR codeword.
4. Overlay and blend the watermarking image on top of the original image.

Correspondingly, the decoder of FIG. 2B decodes the watermarked message X from the screenshot of the watermarked image as follows.
5. Detect and extract QR code words from screenshots.
6. Decode the K-bit binary string from the extracted QR codeword.
7. Decode the watermarking message X from the k-bit binary string and the side information Y.

Note that in many implementations, one or more of the above steps (eg, steps 6-7) may be combined into a single step for better performance.

Note that the QR codeword contains a pattern for detection and an error correction code in 2D layout. In some embodiments, instead of QR codewords, 1D error correction codewords may be used along with 1D patterns for detection for better performance/flexibility in generating watermarking images. Examples of 1D error correcting codes include Reed-Solomon codes, Turbo codes, LDPC (Low Density Parity Check) codes, and other common linear block codes.

Given step 1 in the encoding process above, in order to determine K, we need to know H(X|Y) a priori, but knowledge of the realization of Y (i.e., the actual side-information sequence) is not necessary. Examples of side information Y for which a priori knowledge of H(X|Y) is available include screenshot timestamps, decoding times, and any extrinsic information about the screenshot (e.g. its IP address and geographic location). site information, publisher information, and platform information).

The following description uses screenshot timestamps as an example, but other similar implementations may utilize IP address information and/or platform information, or a combination thereof.

The query ID described in Figure 1B above is a 128-bit Recall that it is a binary string. In a typical application, the screenshot timestamp T _s is strongly correlated with the timestamp T _q in the query ID such that T _q ≤ T _s and with high probability T _s −T _q ≤ Δ There is a non-negative integer Δ.

In view of these, instead of using 64 bits for timestamps in such an implementation, the encoder of FIG. 2B uses K=(ceil(64-log2(Δ)) +64) bits, where Y is T _s and 'ceil' is a ceiling function that rounds its argument up to the nearest integer. Thus, in one embodiment, a binning scheme is used to encode T _q , each bin containing candidate timestamps that are at least Δ microseconds apart, and bin indices of length ceil(64-log2(Δ )) is the suffix of T _q .

The proposed binning scheme is based on the fact that the most significant bits are the same for two timestamps that are close to each other. For example, the timestamp of 2019-01-01 in epoch time is 1546300800 and its binary is
0b0101 0111 1110 0101 1010 0011 0101 1110 0110 0110 0000 0000 0000.

2018-01-01 has a timestamp of 1514764800 and its binary version is
0b0101 0110 0001 1010 1011 1010 1001 1101 0010 1000 0000 0000 0000.

The upper 18 bits of the 64-bit representation are the same. The closer two timestamps are, the more significant bits are the same. In typical implementations, image timestamps and screenshot timestamps may be fairly close, typically within a day, week, or month, and therefore have the same greater number of bits. There are cases.

By using the binning scheme described above, in some implementations the system uses K=(ceil(64-log2(Δ))+64) bits to approximate the most significant bits in T _q . By removing log2(Δ), we can encode the query ID. FIG. 1C is a diagram of a data format 150' for image watermarking, according to one such implementation. As shown, the IP address 154 and process ID 156 are the same as in the implementation of FIG. 1B, but the timestamp is reduced to a fraction of the least significant bits 158, adding an additional Data 160 can be added.

On the decoder side, after extracting the QR code from the received screenshot and obtaining the K-bit binary string from decoding, timestamp LSB 158 may identify the index of the bin containing the correct timestamp T _q . To recover T _q , the decoder can combine the first log2(Δ) bits of T _s with the (64-log2(Δ)) bitbin index to obtain T′ _q . If log2(Δ) is not an integer, here instead the smallest integer greater than log2(Δ) is used, ie ceil(log2(Δ))). Since T _s −T _q ≦Δ with high probability in many implementations, it is highly probable that T′ _q =T _q at the decoder. If T _s −T _q >Δ, m is a positive integer, then T _q must be in the following list of size m, as long as T _s −T _q ≦mΔ.
{ _T'q , _T'q -Δ, _T'q -2Δ,..., _T'q- (m-1)Δ}.

Since (ceil(64-log2(Δ))+64)<128, these implementations effectively reduce the message rate required to recover the query ID at the decoder. Such reduction may then be exploited in two ways to improve the decoding success rate.
1. Increase the correction level of the selected QR Code, for example by including additional parity or error correction bits. or
2. Use a smaller macro QR code (eg macro 17).

Note that a 21×21 QR Code can store up to 152 bits of information, as listed in the following table (Table 1).

By reducing the number of bits from 128 to K=(ceil(64-log2(Δ))+64), the system can utilize higher error correction code (ECC) levels or smaller QR codes for successful decoding. Rates can be improved (eg, change from medium to quartile).

Improving Message Rates The implementations described above take advantage of the side information available at the decoder to improve the robustness of watermarking. From a different perspective, the system can also utilize side information to improve message rates.

In such an implementation, the encoder of FIG. 2B can embed the watermarking message into the image as follows.
1. Convert the 128-bit query ID to a 128-bit binary string with K bits of additional information, where K is determined by H(X)-H(X|Y).
2. Convert a 128-bit binary string to a QR codeword.
3. Generate a watermarking image containing at least one copy of the QR codeword.
4. Overlay and blend the watermarking image on top of the source image.

Correspondingly, the decoder of FIG. 2B may decode the watermarking message X from the screenshot of the image as follows.
4. Detect and extract QR code words from screenshots.
5. Decode the 128-bit binary string from the extracted QR codeword.
6. Decode the 128-bit query ID with K bits of additional information from the 128-bit binary string and side information Y.

Compared to systems that do not implement these methods, these implementations provide additional K-bit messaging capability essentially for free, ie, with the same decoding success rate and same distortion level. These additional Kbits may be used to provide better tracking capabilities and/or user experience from a usability perspective.

While discussed above primarily in terms of reducing the data size of timestamps within the watermark data, similar implementations can be used with binning applied to IP addresses and/or process identifiers. can. For example, if typical process identifiers are all less than 20 bits long, 12 bits may be removed from the process ID 156 MSB. Similarly, part of the IP address in the watermark data (e.g. the leftmost 8 bits) may be derived from side information available at the decoder (e.g. the IP address used to submit the screenshot). addresses, decoder IP addresses, etc.). Combinations of these fields can be processed in this way to further reduce the data size.

FIG. 3 is a block diagram of a system for image watermarking, according to some implementations. Client devices 300, which can include desktop computers, laptop computers, tablet computers, wearable computers, smart phones, embedded computers, smart cars, or any other type and form of computing device, communicate one It may communicate with one or more servers 314 .

In many implementations, client device 300 may include processor 302 and memory device 306 . Memory device 306 may store machine instructions that, when executed by a processor, cause the processor to perform one or more of the operations described herein. Processor 302 may include a microprocessor, ASIC, FPGA, etc., or a combination thereof. In many implementations, the processor may be a multi-core processor or an array of processors. Memory device 306 may include, without limitation, electronic, optical, magnetic, or any other storage device capable of providing program instructions to a processor. A memory device may be a floppy disk, CD-ROM, DVD, magnetic disk, memory chip, ROM, RAM, EEPROM, EPROM, flash memory, optical media, or any other suitable memory from which a processor can read instructions. can contain. Instructions may include code from any suitable computer programming language such as, but not limited to, C, C++, C#, Java, JavaScript, Perl, HTML, XML, Python, and Visual Basic.

Client device 300 may include one or more network interfaces 304 . Network interface 304 can be any of a variety of 802.11 wireless, such as Ethernet, including 10 Base T, 100 Base T, or 1000 Base T ("Gigabit"), 802.11a, 802.11b, 802.11g, 802.11n, or 802.11ac. cellular, including CDMA, LTE, 3G or 4G cellular, Bluetooth or other short-range wireless connections, or any combination of these or other interfaces for communicating with a network can include In many implementations, client device 300 may include multiple network interfaces 304 of different types that allow connection to various networks 312 . Correspondingly, network 312 may be a local area network (LAN), a wide area network (WAN) such as the Internet, a cellular network, a broadband network, a Bluetooth network, an 802.11 (WiFi) network, a satellite network, or any of these or other networks. Any combination may be included and may include one or more additional devices (eg, routers, switches, firewalls, hubs, network accelerators, caches, etc.).

A client device may include one or more user interface devices. A user interface device communicates data to a user by generating sensory information (e.g., visualization on a display, one or more sounds, tactile feedback, etc.) and/or interprets sensory information received from a user. It may be any electronic device that converts it into an electronic signal (eg, keyboard, mouse, pointing device, touch screen display, microphone, etc.). One or more user interface devices may be internal to the housing of the client device, such as a built-in display, touch screen, microphone, or a monitor connected to the client device, connected to the client device, according to various implementations. may be external to the housing of the client device, such as a connected speaker.

Memory 306 may contain applications 308 for execution by process 302 . Applications 308 may include any type and form of application such as a media application, web browser, productivity application, or any other such application. Application 308 can receive images from a content server that include watermarks embedded within the images, and can display them via a user interface for the user of the client device.

Memory 306 may also include capture engine 310, which may be part of application 308 (eg, a browser plug-in or extension) and/or part of the device's operating system. Capture engine 310 may include an application, server, service, daemon, routine, or other executable logic for capturing screenshots of rendered images that include watermarks. Capture engine 310 may be configured to capture screenshots of all images or some images. For example, in some implementations, the capture engine 310 responds to image metadata or to scripts executed by the application 308 (e.g., scripts embedded in web pages displayed by a browser). ), it can be triggered to take a screenshot of the image. The capture engine 310, in some implementations, can take screenshots of images only, or can take screenshots of the entire display or screen. In further implementations, the capture engine can crop the captured image to just the desired image. This can be done, for example, based on the coordinates of the representation of the image within the display. Capture engine 310 may add metadata to the screenshot, such as capture time (eg, epoch time), as described above. Capture engine 310 may send screenshots to a monitoring server via network interface 304 . In some implementations, the capture engine 310 may include scripts embedded in a webpage and executed by the application 308 while rendering the webpage, such webpages being captured by the capture engine. It may also contain embedded images or links to images to import.

Server 314 may include content servers and/or monitoring servers, which may be the same device or different devices. Server 314 may include one or more processors 302 , network interfaces 304 , and memory devices 306 . A content server 314 may include one or more content items 316 in storage, such as watermarking images, as well as other content (eg, web pages, other media, etc.). Content server 314 may also include encoder 202, as described above in connection with FIGS. 2A and 2B. Encoder 202 may include software, hardware, or a combination of hardware and software. For example, encoder 202 may include an ASIC, FPGA, or other specialized hardware for embedding watermarks in images.

The monitoring server may include decoder 204, as described above in connection with FIG. 2B. Decoder 204 may include software, hardware, or a combination of hardware and software. For example, decoder 204 may include an ASIC, FPGA, or other specialized hardware for identifying and decoding watermarks from images. As mentioned above, the decoder 204 may receive side information to aid in decoding the watermark, such as the screenshot time from the screenshot metadata received from the capture engine 310 .

FIG. 4 is a flowchart of a method for image watermarking, according to some implementations. At step 402, a client device may request a content item. A request may be triggered during rendering of a web page by a browser or other application (eg, an interstitial content item during a break in a mobile game, or any other type and form of content). At step 404, the content server 314 may select a content item. Content items may be selected via any means and may be based on client device type, user account or device identifier, contextual items within a web page or other application, or any other such information.

At step 406, the content server 314 may include one or more identifiers, including a timestamp, a server identifier or server IP address, and/or a process identifier of the process used to select the content item. A mark identifier can be generated. In some implementations, the watermark identifier may include additional information such as the identifier of the content item. At step 408, the content item may be encoded with a watermark. As noted above, encoding a content item includes an alpha channel with pixels modified from default values or patterns representing modified bits of the encoded watermark (e.g., QR code or similar code). generating the overlay with . The watermark may be repeated at predetermined intervals or spaces across the image. The overlay can then be blended or combined with the image to generate the encoded content item. At step 410, the encoded content item may be transmitted by the content server to the client device.

Although it is shown that the content server generates the watermark identifier and encodes the watermark after receiving the request for the content item, in some implementations the content item is (e.g., before step 402 ) pre-encoded, the content server may select pre-encoded content items for distribution. In many implementations, such pre-encoding may be performed within a predetermined time frame prior to the request. For example, a content item may be encoded with a given timestamp, utilized for a predetermined period of time (eg, two weeks), and then replaced with a new timestamp or re-encoded. This may allow the content server to perform the encoding process during less busy hours while still ensuring that the content and timestamps are relatively new. As mentioned above, the shorter the window in which the pre-encoded content item can be used, the more data can be encoded into the watermark and/or the more robust the watermark can be, but in the above example Even if there is, windows of one year or more can be used while still greatly reducing the data required.

At step 412, the client device may render the content item within an application such as, for example, a web browser, media player, game, or other application. At step 414, the client device's capture engine (running as a separate service or as an application plug-in or script) may capture a screenshot of the content item. The screenshot may be cropped or restricted to the content item, or it may be full screen or part of the screen. Screenshots can be identified via metadata with a capture timestamp, and can include other identifiers (eg, device identifiers, application and/or web page context identifiers, etc.). In other implementations, the capture timestamp may be provided via other means. For example, in some implementations, given that the screenshot capture time and transmission time to the server may be very close (e.g., within a few seconds), the packet transmission time (e.g., transport layer header time (Identified or extracted from the header of the packet, such as the Stamp Options field) or the time of receipt can be utilized as the capture timestamp. At step 416, the client device may send the screenshot to the monitoring server.

In step 418, the content server or the monitoring server, which may be a different device, may receive the screenshot, and in some implementations may extract the timestamp from the metadata of the screenshot, or It may identify the time the screenshot was sent or received. The timestamp may be provided as side information to the decoder of the monitoring server.

At step 420, the decoder may scan the screenshot and extract any identified watermarks. In some implementations, where a watermark appears multiple times in a screenshot, the decoder compares the identified watermarks, and the watermark with the least distortion (e.g., the watermark with the highest number of other watermarks in the image). matching watermarks, watermarks that are averages of other identified watermarks, etc.) can be selected or generated. At step 422, the decoder may convert the watermark to a string.

At step 424, the decoder may generate a timestamp from a portion of the timestamp extracted from step 418 (eg, a predetermined number of least significant bits), and use the generated timestamp to construct the string. Decoding can be tested (e.g. using the generated timestamp to apply an error correction algorithm to the decoded string). If the string decodes correctly according to the error correction bits, in step 426 the monitoring server can process the screenshot image or data associated with the content item (e.g., identify the content server via IP address and process identifier). identifying, comparing screenshots to original Content Items to detect rendering distortions or corruption, tracking delivery of Content Items, etc.).

If the string is not decoded correctly, the decoder can advance the timestamps generated according to the value Δ from the binning scheme and retest the decoding at step 424 . This is done until the decryption is successful or all of the bin index values have been tested (either because the watermark is corrupted or improperly extracted, or the content item from before the usage time window mentioned above). is used), may be repeated iteratively. If all bin index values have been tested and the decoding is unsuccessful, then at step 428 the decoder may report the error to an administrator or user of the system.

Accordingly, the systems and methods described herein provide improved image watermarking for improved robustness and capacity without reducing perceptibility. Specifically, the systems and methods described herein enable higher decoding success rates at the same distortion level and message rate, or higher message rates at the same distortion level and decoding success rate. enable. Implementations of these systems take advantage of sidechains of additional information available only to the decoder, not the encoder, to achieve asymptotically lossless data compression, so that the same message is sent with fewer bits. enable

Implementations of the subject matter and operations described herein may be implemented using either digital electronic circuitry or computer software, firmware, or hardware including the structures disclosed herein and their structural equivalents, or any one thereof. can be implemented in one or more combinations. Implementations of the subject matter described herein are encoded on one or more computer storage media for execution by or for controlling the operation of a data processing apparatus. It can be implemented as one or more computer programs or modules of computer program instructions. Alternatively, or additionally, the program instructions may be implemented in an artificially generated propagated signal generated to encode information for transmission to appropriate receiver equipment for execution by data processing equipment. , for example, can be encoded on a machine-generated electrical, optical, or electromagnetic signal. A computer storage medium may be or be included in a computer readable storage device, a computer readable storage substrate, a random or serial access memory array or device, or a combination of one or more thereof. Further, although a computer storage medium is not a propagated signal, a computer storage medium can be a source or destination of computer program instructions encoded in an artificially generated propagated signal. A computer storage medium can also be or be contained in one or more separate components or media (eg, multiple CDs, disks, or other storage devices). As such, a computer storage medium may be tangible.

The operations described herein may be implemented as operations performed by a data processing apparatus on data stored in one or more computer readable storage devices or received from other sources. can be done.

The term "client" or "server" includes all kinds of apparatus, devices and machines for processing data, such as programmable processors, computers, systems-on-chips, or more or combinations of the above. The device may include dedicated logic circuits such as FPGAs (Field Programmable Gate Arrays) or ASICs (Application Specific Integrated Circuits), for example. The apparatus comprises, in addition to hardware, code that creates an execution environment for the computer program in question, such as processor firmware, protocol stacks, database management systems, operating systems, cross-platform runtime environments, virtual machines, or one of these. Codes comprising one or more combinations may also be included. Devices and execution environments can implement a variety of different computing model infrastructures, such as web services, distributed computing, and grid computing infrastructures.

A computer program (also called a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and as a stand-alone program, Or may be arranged in any form, such as as modules, components, subroutines, objects, or other units suitable for use in a computing environment. A computer program may, but need not, correspond to a file in a file system. A program may be written either in a single file dedicated to the program, or in multiple coordinated files (e.g., files storing one or more modules, subprograms, or portions of code) (e.g., markup It can be stored in part of a file that holds other programs or data, such as one or more scripts stored in a language document. A computer program can be deployed to be executed on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a communication network.

The processes and logic flows described herein are one or more programmable programs that run one or more computer programs to perform actions by operating on input data and generating output. It can be executed by a processor. The processes and logic flows may also be performed by dedicated logic circuits such as FPGAs (Field Programmable Gate Arrays) or ASICs (Application Specific Integrated Circuits), and the device may be implemented as dedicated logic circuits.

Processors suitable for the execution of a computer program include both general and special purpose microprocessors, and one or more processors of any kind of digital computer. Generally, a processor receives instructions and data from read-only memory and/or random-access memory. The essential elements of a computer are a processor for performing actions according to instructions, and one or more memory devices for storing instructions and data. Generally, a computer also includes one or more mass storage devices, such as, for example, magnetic, magneto-optical or optical disks, for storing data on or receiving data from one or more mass storage devices. , transfer data to it, or both. However, a computer need not have such devices. Additionally, the computer may be connected to another device such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or It can be embedded in a portable storage device (such as a Universal Serial Bus (USB) flash drive). Devices suitable for storing computer program instructions and data include, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices, magnetic disks such as internal hard disks or removable disks, magneto-optical disks, and CD-ROMs. Includes all forms of non-volatile memory, media and memory devices including ROM and DVD-ROM discs. The processor and memory may be supplemented by or incorporated in dedicated logic circuitry.

To provide user interaction, implementations of the subject matter described herein include, for example, CRTs (Cathode Ray Tubes), LCDs (Liquid Crystal Displays), OLEDs (Organic Light Emitting Diodes), TFTs (Thin Film Transistors), Plasma , other flexible configurations, or any other display device for displaying information to a user, such as a monitor, as well as a keyboard, pointing device, e.g., mouse, trackball, etc., through which the user can provide input to the computer Or it can be implemented on a computer with a touch screen, touch pad, or the like. Other types of devices may also be used to provide interaction with the user, and the feedback provided to the user may be any form of sensory feedback, e.g., visual, auditory, or tactile feedback. and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, the computer sends and receives documents to and from the device used by the user by sending web pages to the web browser on the user's client device in response to requests received from the web browser. , can interact with the user.

Implementations of the subject matter described herein include backend components, such as data servers, or middleware components, such as application servers, or client computers having, for example, graphical user interfaces, or user including a front-end component such as a client computer with a web browser capable of interacting with an implementation of the subject matter described herein, or one or more such back-end, middleware, or front-end components can be implemented in a computing system comprising any combination of The components of the system can be interconnected by any form or medium of digital data communication, eg, a communication network. Communication networks can include local area networks (“LAN”) and wide area networks (“WAN”), internetworks (eg, the Internet), and peer-to-peer networks (eg, ad-hoc peer-to-peer networks).

In situations where the systems described herein may collect or utilize personal information about the user, the user may collect personal information (e.g., the user's social networks, social behavior or activities, the user's preferences, or the opportunity to control programs or features that may collect information about a user's location, or whether or how content is received from a content server or other data processing system that may be more relevant to the user; may be given the opportunity to control how they receive content. In addition, some data may be anonymized in one or more ways before it is stored or used, thus removing personally identifiable information when generating parameters. . For example, a user's identity may be anonymized so that no personally identifiable information can be determined about the user, or the user's geographic location may be generalized, and the user's Location information (such as city, zip code, or state level) is obtained so that a specific location cannot be determined. Thus, users can control how information is collected about them and how it is used by content servers.

While this specification contains many specific implementation details, these are not limitations on the scope of any invention or that may be claimed, but rather are specific to particular implementations of particular inventions. shall be construed as a description of the characteristics of Certain features that are described in this specification in the context of separate implementations can also be implemented in combination in a single implementation. Conversely, various features that are described in the context of a single implementation can also be implemented in multiple implementations separately or in any suitable subcombination. Further, although features are described above and originally claimed as working in some combination, one or more features from the claimed combination may be used in some combinations. Cases can be cut from a combination, and a claimed combination may cover a subcombination, or a variation of a subcombination.

Similarly, although acts have been shown in the figures in a particular order, it does not mean that such acts are performed in the specific order shown, or in a sequential order to achieve the desired result. It should not be understood as requiring that all illustrated acts be performed for purposes of illustration. Multitasking and parallel processing may be advantageous in some situations. Furthermore, the separation of various system components in the implementations described above is not to be understood as requiring such separation in all implementations, and the program components and systems described generally can be combined into a single system. can be incorporated together into multiple software products, or can be packaged into multiple software products.

Accordingly, specific implementations of the subject matter have been described. Other implementations are within the scope of the following claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve desirable results. Additionally, the processes illustrated in the accompanying drawings do not necessarily require the particular order shown or sequential order to achieve desirable results. In some implementations, multitasking or parallel processing can be utilized.

100 images
102 watermark code
150 data format
152 Timestamp
154 IP addresses
156 process identifier
158 Timestamp LSB
160 additional data
200 systems
202 Encoder
204 decoder
206 Image "S"
208 Message "X"
210 Watermarked image "S'"
212 communication channels
214 Side information "Y"
300 client devices
302 processor
304 network interface
306 memory device
308 applications
310 capture engine
312 network
314 servers
316 content items

Claims (27)

A decoder for a watermarking system, comprising:
from a client device, (i) a screen shot of an image displayed by said client device, said image being watermarked with a string associated with image metadata associated with said image; receiving a packet containing a shot and (ii) screenshot metadata associated with said screenshot of said image;
extracting the screenshot metadata and the string from the packet;
decoding an identifier associated with the image from the string using a portion of the screenshot metadata, the identifier comprising image metadata;
and tracking delivery of a content item associated with said image in response to said decoded identifier. 2. A decoder for watermarking system according to claim 1, wherein said screenshot metadata comprises a timestamp of said screenshot of said image, said image metadata comprises a timestamp of said image. 3. The decoder of claim 2, wherein the decoder is configured to extract the timestamp of the screenshot of the image from a header of the packet containing the screenshot of the image and the screenshot metadata. Decoder for the watermarking system described. 4. A decoder for a watermarking system according to claim 2 or 3, wherein said string comprises a subset of said time stamps of said image. the decoder is configured to decode the identifier from the string by concatenating a portion of the timestamp of the screenshot of the image with the subset of the timestamp of the image; A decoder for a watermarking system according to claim 4. 6. The string of claims 4 or 5, wherein the string comprises a number of error correction bits greater than the difference between the length of the timestamp of the image and the length of the subset of timestamps of the image. Decoder for the watermarking system described in . 7. The decoder according to any one of claims 1 to 6, wherein said decoder is arranged to decode said identifier from said string by combining said part of said screenshot metadata with a predetermined offset. Decoder for the watermarking system described. The decoder is configured to decode the identifier from the string by iteratively combining the portion of the screenshot metadata with multiples of the predetermined offset until successfully decoding the identifier. 8. A decoder for watermarking system according to claim 7, wherein: Decoder for a watermarking system according to any one of claims 1 to 8, wherein said string contains an address of a content server that generated said image watermarked with said string. 10. A decoder for a watermarking system according to claim 9, wherein said string comprises an identifier of said content server process that generated said image watermarked with said string. A method for watermarking performed by a computing device, comprising:
(i) a screenshot of an image displayed by said client device, said image being watermarked with a string associated with image metadata associated with said image; and (ii) screenshot metadata associated with said screenshot of said image;
extracting the screenshot metadata and the string from the packet by the decoder;
decoding, by the decoder, an identifier associated with the image from the string using a portion of the screenshot metadata, the identifier comprising image metadata;
tracking, by said decoder, delivery of a content item associated with said image in response to said decoded identifier. 12. The method for watermarking according to claim 11, wherein said screenshot metadata comprises a timestamp of said screenshot of said image, said image metadata comprises a timestamp of said image. 13. The method for watermarking according to claim 12, further comprising extracting, by the decoder, the timestamp of the screenshot of the image from a header of the packet containing the screenshot of the image. 14. A method for watermarking according to claim 12 or 13, wherein said string comprises a subset of said time stamps of said image. 15. The method of claim 14, wherein decoding the identifier from the string further comprises concatenating a portion of the timestamp of the screenshot of the image with the subset of timestamps of the image. A method for watermarking. Claim 14 or claim 15, wherein said string comprises a number of error correction bits greater than the difference between the length of said timestamp of said image and the length of said subset of said timestamps of said image. A method for watermarking as described in . 17. Watermarking according to any one of claims 11 to 16, wherein decoding the identifier from the string further comprises combining the portion of the screenshot metadata with a predetermined offset. way for. 3. The step of decoding the identifier from the string further comprises iteratively combining the portion of the screenshot metadata with multiples of the predetermined offset until the identifier is successfully decoded. A method for watermarking according to 17. 19. A method for watermarking according to any one of claims 11 to 18, wherein said string comprises an address of a content server that generated said image watermarked with said string. 20. The method for watermarking of claim 19, wherein the string includes an identifier of the content server process that generated the image watermarked with the string. A computer readable storage medium containing instructions which, when executed by a computing device, cause said computing device to perform the method of any one of claims 11 to 20. receiving an image and image metadata associated with the image;
generating a string from a subset of the image metadata;
encoding a watermark from the string;
embedding said watermark in said image; and
a decoder of said device or a second device,
from a client device, (i) a screen shot of said image displayed by said client device, said image being watermarked with said string generated from said subset of said image metadata; receiving a packet containing a shot and (ii) screenshot metadata associated with said screenshot of said image;
extracting the string and additional screenshot metadata from the packet containing the screenshot of the image displayed by the client device and the screenshot metadata;
decoding an identifier associated with the image from the string using a portion of the additional screenshot metadata, the identifier including a portion of the image metadata; and,
tracking delivery of a content item associated with the image in response to the decoded identifier;
watermarking system. 23. The watermarking system of claim 22, wherein the image metadata includes a timestamp of the image and the additional screenshot metadata includes a timestamp of the screenshot of the image by the client device. 24. A watermarking system according to claim 22 or 23, wherein said encoder of said device is configured to generate said string from a predetermined number of least significant bits of said image metadata. A method for watermarking performed by a computing device, comprising:
receiving, by an encoder of a device, an image and image metadata associated with said image;
generating, by the encoder, a string from the subset of the image metadata;
encoding a watermark from the string by the encoder;
embedding said watermark in said image by said encoder, wherein a decoder of said device or a second device comprises:
from a client device, (i) a screen shot of said image displayed by said client device, said image being watermarked with said string generated from said subset of said image metadata; receiving a packet containing a shot and (ii) screenshot metadata associated with said screenshot of said image;
extracting the string and additional screenshot metadata from the packet containing the screenshot of the image displayed by the client device and the screenshot metadata;
decoding an identifier associated with the image from the string using a portion of the additional screenshot metadata, the identifier including a portion of the image metadata; and,
tracking delivery of a content item associated with the image in response to the decoded identifier;
A method for watermarking. 26. The watermarking of claim 25, wherein the image metadata comprises a timestamp of the image and the additional screenshot metadata comprises a timestamp of the screenshot of the image by the client device. way for. 27. A method for watermarking according to claim 25 or 26, further comprising generating said string from a predetermined number of least significant bits of said image metadata.

Images