KR102578027B1 - Improved image watermarking - Google Patents

Abstract

The present disclosure provides improved systems and methods for image watermarking to improve robustness and capacity without compromising perceptibility. In particular, the systems and methods discussed herein may allow for higher decoding success rates at the same distortion level and message rate or may allow for higher message rates at the same distortion level and decoding success rate. Implementations of these systems achieve asymptotic lossless data compression by exploiting an additional chain of additional information that is only available at the decoder and not at the encoder, allowing the same message to be transmitted with fewer bits.

Description

Improved image watermarking

This application claims the benefit of priority to PCT Application No. PCT/US2019/037959, filed June 19, 2019, entitled “Improved Image Watermarking,” the entirety of which is incorporated herein by reference.

Image watermarking is a technology for embedding visually imperceptible data or messages into images, and can be classified as non-blind or blind, respectively, depending on whether the original image is required for watermark extraction. Blind watermarking is particularly useful because embedded data can be recovered without access to the original pre-embedded image.

However, current implementations of blind image watermarking are limited in terms of perceptibility (e.g., whether distortions due to the watermarking message embedding can be detected by a viewer), robustness (e.g., the success rate of the decoder in decoding the embedded message), and ) and capacity (e.g., the amount or speed at which data can be embedded in an image) may be an issue. In many implementations, increasing one of these can cause a drastic performance degradation in the other.

The systems and methods discussed herein provide improved image watermarking to improve robustness and capacity without compromising perceptibility. In particular, the systems and methods discussed herein allow for higher decoding success rates at the same distortion level and message rate or higher message rates at the same distortion level and decoding success rate. Implementations of these systems achieve asymptotic lossless data compression by leveraging side chains (or side channels) of additional information that is only available at the decoder and not at the encoder, allowing the same message to be transmitted more robustly or with fewer bits. do.

In one aspect, the present disclosure relates to a system for improved watermarking. This system includes a decoder on the device. The decoder receives a capture of an image that includes at least one embedded watermark, determines a timestamp of the capture, decodes a binary string from the embedded watermark, and uses a portion of the timestamp of the capture to create a time stamp of the image. Decode an identifier from a binary string containing a timestamp; It is configured to output a decoded identifier.

In some implementations, the timestamp of the capture is identified in the capture's metadata. In some implementations, the decoder is configured to extract the timestamp of the capture from the header of the packet containing the capture. In some implementations, the binary string of the embedded watermark contains a subset of the image's timestamp. In a further implementation, the decoder is configured to decode the identifier from the binary string by concatenating a portion of the timestamp of the capture with a subset of the timestamp of the image. In another further implementation, the binary string of the embedded watermark includes a number of error correction bits that are greater than the difference between the length of the image's timestamp and the length of a subset of the image's timestamps.

In some implementations, the decoder is configured to decode an identifier from a binary string by combining a portion of the capture's timestamp with a predetermined offset. In a further implementation, the decoder is configured to decode an identifier from a binary string by repeatedly combining a portion of a timestamp of the capture with a multiple of a predetermined offset until successfully decoding the identifier.

In some implementations, the binary string contains the address of the content server that generated the image containing at least one embedded watermark. In a further implementation, the binary string includes a process identifier of the content server that generated the image containing at least one embedded watermark.

In another aspect, the present disclosure relates to a method for improved watermarking. The method includes receiving, from a client device, a capture of an image including at least one embedded watermark, by a decoder of the device. The method also includes determining, by the decoder, a timestamp of capture. The method also includes decoding, by a decoder, the binary string from the embedded watermark. The method also includes decoding, by a decoder, an identifier from a binary string containing the timestamp of the image, using a portion of the timestamp of the capture. The method also includes outputting, by the decoder, a decoded identifier.

In some implementations, the timestamp of the capture is identified in the capture's metadata. In some implementations, the method includes extracting, by a decoder, a timestamp of the capture from a header of a packet containing the capture. In some implementations, the binary string of the embedded watermark contains a subset of the image's timestamp. In a further implementation, the method includes associating a portion of the timestamp of the capture with a subset of the timestamp of the image. In another further implementation, the binary string of the embedded watermark includes a number of error correction bits that are greater than the difference between the length of the image's timestamp and the length of a subset of the image's timestamps.

In some implementations, the method includes combining a portion of the time stamp of the capture with a predetermined offset. In a further implementation, the method includes repeatedly combining a portion of the time stamp of the capture with a multiple of a predetermined offset until successfully decoding the identifier.

In some implementations, the binary string contains the address of the content server that generated the image containing at least one embedded watermark. In a further implementation, the binary string includes an identifier of the process of the content server that created the image containing at least one embedded watermark.

In another aspect, the present disclosure relates to a watermarking system. The system receives images and metadata associated with the images; generate a binary string from a subset of metadata associated with the image; An encoder on the device configured to encode the watermark from the binary string and embed the watermark in the image. The device and the decoder on the second device recover metadata associated with the image from a subset of metadata associated with the image encoded in the embedded watermark and additional metadata associated with a display capture of the image on the third device.

In some implementations, the metadata associated with the image includes a timestamp of the image, and the additional metadata includes a timestamp of display capture of the image on the third device. In some implementations, the encoder of the device is configured to generate a binary string from a predetermined number of least significant bits of metadata associated with the image.

In another aspect, the present disclosure relates to a watermarking method. The method includes receiving, by an encoder on a device, an image and metadata associated with the image. The method also includes generating, by the encoder, a binary string from a subset of metadata associated with the image. The method also includes encoding the watermark from the binary string, by an encoder. The method also includes embedding a watermark in the image, by the encoder. The device or a decoder on the second device recovers metadata associated with the image from a subset of metadata associated with the image encoded in the embedded watermark and additional metadata associated with a display capture of the image on the third device.

In some implementations, the metadata associated with the image includes a timestamp of the image, and the additional metadata includes a timestamp of display capture of the image on the third device. In some implementations, the method includes generating a binary string from a predetermined number of least significant bits of metadata associated with the image.

The disclosure also provides a computer program that, when executed by a computing device, includes instructions that cause the computing device to perform any of the methods disclosed herein. The disclosure also provides a computer-readable medium containing instructions that, when executed by a computing device, cause the computing device to perform any of the methods disclosed herein.

Optional features of one aspect may be combined with any other aspect.

Details of one or more implementations are set forth in the accompanying drawings and the description below. Other features, aspects and advantages of the present disclosure will become apparent from the following description, drawings and claims:
1A is an illustration of an example implementation of image watermarking.
Figure 1B is an example of a data format for image watermarking according to one implementation.
Figure 1C is an example of a data format for image watermarking according to another implementation. class
Figure 2A is a block diagram of a system for image watermarking according to one implementation.
Figure 2b is a block diagram of a system for image watermarking according to another implementation.
Figure 3 is a block diagram of a system for image watermarking according to some implementations.
4 is a flow diagram of a method for image watermarking according to some implementations.
Like reference numbers and designations in the various drawings identify similar elements.

Image watermarking is a technology that embeds visually imperceptible data or messages into an image, and can be classified as non-blind or blind depending on whether the original image is required for watermark extraction. Blind watermarking is particularly useful because embedded data can be recovered without access to the original pre-embedded image.

For example, referring briefly to Figure 1A, an example implementation of image watermarking for image 100 is shown. The small watermark code 102 may include an array of pixels sized and placed within the image 100 so as not to be visible to the viewer. As shown, the watermark code 102 may be duplicated throughout the image to provide resistance to local artifacts due to cropping, compression or other corruption, or other such distortions. Although only a few pixels are shown for clarity, in many implementations, the watermark code may include an area of 64 pixels, 128 pixels, or any other amount. Adjusting pixels to define encoding values can be relatively unnoticeable, unlike simple black and white pixels. For example, in many implementations, the pixels that make up the encoded region may have colors that match or are similar to surrounding pixels, but with adjusted alpha (transparency) values. For example, encoding may change a pixel with an alpha value of 0 to an alpha value of 10, 50, 100, 255, or other such values. In some implementations, the code may be detected by identifying pixels with alpha values that are significantly different from surrounding alpha values. In some implementations, differential encoding may be applied by overlay-encoding each bit by changing the alpha value of the pixels within the overlay to encode a different value.

Any type of data may be encoded within watermark 102. Referring briefly to FIG. 1B, a data format 150 for image watermarking according to one implementation is shown. The data format shown includes 128 bits with a 64-bit time stamp 152 (e.g., based on epoch time), IP address 154, and process identifier 156. Data in data format 150 may be referred to herein as a query ID. Many implementations also include error correction bits (not shown) to improve decoding of the watermark. For example, this code can be encoded into a QR code using a Reed-Solomon error correction code embedded within the mark.

In one such implementation, data may be encoded into the image by the content server prior to providing the image to the client device, along with the IP address of the content server and the process identifier of the process that created the image. Then, when the image is received and rendered by the client device, the monitoring process of the client device can capture a screenshot of the image and provide the screenshot to the content server or monitoring server. For example, a monitoring process on a client device may not have access to the image itself (e.g., the image may be stored in a memory location on the client device that the monitoring process cannot access), but (e.g., the frame buffer You can capture a screenshot of an image (by reading the image data from or capturing the image with a camera). The server can decode the watermark to identify the original creation process and server, as well as the time the image was created or marked, and compare the screenshot image to the original image. This allows the system to automatically identify distortions or image damage caused by the rendering or encoding process for the image, as well as identifying different aspects of the image. In implementations where the content server and the monitoring server are different, this may, among other things, allow the monitoring server to identify a particular content server among a plurality of content servers that provided images to the client device. This can be useful for logging, tracking, and analysis, and can be much easier than retrieving HTTP logs or similar logs from a client device (which may not have access to the monitoring server).

Watermarking efficiency is measured between perceptibility, sometimes referred to as "D" (e.g. whether the distortion introduced by embedding the watermarking message can be detected by the viewer), and robustness, sometimes referred to as "E" (e.g. , the success rate of a decoder in decoding an embedded message) and capacity (e.g., the amount or rate at which data can be embedded in an image), sometimes referred to as "R". In many implementations, it may be desirable to have low perceptivity, high robustness, and high capacity. However, in many implementations, improving one of these can lead to drastic performance degradation in the other implementation. For example, to add more data to a message while maintaining robustness, the size of the watermark must be enlarged so that it can be better perceived. Similarly, mark size can be maintained while adding data by removing error correction bits, but this naturally makes the mark more difficult to decode and more susceptible to corruption.

Figure 2A is a block diagram of a system 200 for image watermarking according to one implementation. The system may include an encoder 202 and a decoder 204, which may be on the same or different computing devices (e.g., a content server and a monitoring server). Image (“S”) 206 may be encoded by encoder 202 with message (“X”) 208 to produce a watermarked image (“S”’) 210 containing S+X. there is. The encoded or watermarked image S' may be transmitted via communication channel 212 to, for example, a client device. A corresponding watermarked image (e.g., from a screen shot as discussed above) may be provided to decoder 204. For example, a client device may transmit a watermarked image to decoder 204 over communication channel 212. The communication channel may therefore include any combination of networks and devices between the encoder and decoder, potentially introducing any kind of additional distortion. For example, channels may be lost as a result of intentional or unintentional attacks or compromises. Examples of unintentional corruption include image rotation, resizing (scaling), and format conversion. Examples of intentional corruption include noise injection (e.g., adding information) and attempts to remove watermarking code (e.g., subtracting information).

The decoder 204 can detect and decode the watermark from the watermark image (S') to recover the original message (X) 208', and apply error correction as needed (and potentially multiple of watermarks and compares each decoded message to rule out errors or distortions in a single watermark.

Accordingly, the encoder encodes a message, such as a timestamp/address/process ID string discussed above, along with an arbitrary error correction code, into a mark, such as a QR code, and, through blending of the alpha channel overlay, at least one copy of the mark. can be encoded into the image, and the decoder can decode the message by detecting inconsistencies that identify the overlay pattern and QR code, decode the original string, and identify the original timestamp/address/process ID. .

Although these systems are relatively successful, they have high error rates. In an experiment decoding embedded marks from screenshots of encoded images, the decoding success rate was 44.03%.

As mentioned above, given a fixed message rate (e.g. 128 bits), the factors that affect the decoding success rate are the distortion introduced to the image by the encoder (D _e ) and the Distortion between the watermarked images at the output (D _c ). In general, the robustness of image watermarking measured from the decoding success rate is controlled by D _e , and for the same D _c , if D _e increases, the decoding success rate can be higher. However, in most cases, the watermark should not be visually perceptible in the watermarked image. This requirement imposes an upper bound on D _e . This constraint on D _e essentially means an upper bound on the decoding success rate for any given channel. In some extreme cases where D _c introduced by the channel is large, the decoding success rate may drop to almost zero, thus limiting the applicability of this watermarking implementation.

(D ₀ , E ₀ , R ₀ ) represent the distortion, decoding success rate and message rate of the implementation of the watermarking method discussed above, respectively. In a typical implementation, an improvement in one of the three quantities inevitably results in a performance loss in at least one of the other quantities. For example, to improve E ₀ , D ₀ must be sacrificed while maintaining R ₀ or R ₀ must be reduced while maintaining D ₀ . However, in many applications, both D ₀ and R ₀ currently have strict constraints: D ₀ must be upper bounded to avoid negatively impacting the user experience, and R ₀ must be upper bounded so that the watermarking message is useful for tracking purposes. It must be specified as the lower limit. In this context, current implementations of watermarking leave little room for improving E ₀ .

The systems and methods discussed herein provide improved image watermarking to improve robustness and capacity without compromising perceptibility. In particular, the systems and methods discussed herein enable higher decoding success rates at the same distortion level and message rate or enable higher message rates at the same distortion level and decoding success rate. Implementations of these systems achieve asymptotic lossless data compression by exploiting an additional chain of additional information that is only available at the decoder and not at the encoder, allowing the same message to be transmitted with fewer bits.

Because distortion constraints are given by the application, the systems discussed herein focus on the trade-off between decoding success rate and message rate. In particular, the system avoids the above-mentioned lower limit on message rate without compromising the usefulness of watermarking messages. As a result, this provides greater flexibility in finding the right compromise between robustness and capacity that was not possible with previous implementations. In particular, side information available only at the decoder can be used to achieve asymptotic lossless compression.

Figure 2B is a block diagram of a system 200' for image watermarking according to one such implementation. As discussed in relation to FIG. 2A , encoder 202 encodes image 206 with message 208 to generate watermarked image 210 that can be provided to decoder 204 over communication channel 212. ) is encoded. However, to recover the original message 208', the decoder uses additional side information ("Y") 214 that is not available to the encoder. This eliminates the need for separate communication between the encoder and decoder, and can be particularly advantageous in implementations where the content server and monitoring server are not the same device (and may not be controlled by the same entity).

Between Figures 2a and 2b, the main difference is the introduction of side information (Y) in the decoder of Figure 2b. From the classical source coding theorem, the minimum rate required for lossless recovery of message (X) in the decoder in Figure 2a is given by the marginal entropy of X (X). Correspondingly, from the SlepianWolf coding theorem, the minimum speed required for lossless recovery of message (X) in the decoder in Figure 2b is given by the conditional entropy of Since H(X|Y) ≤ H(X) for any ( It is possible to use speed. The stronger the correlation between X and Y, the lower the message rate can be.

Improved robustness

In a first implementation, the system may utilize side information (Y) at the decoder to improve robustness. In some such implementations, the encoder in Figure 2b embeds a watermarking message in the image as follows:

1. Convert the watermarking message (X) to a K-bit binary string, where K is determined by H(X|Y).

2. Convert the K-bit binary string to QR codeword.

3. Create a watermarked image containing at least one copy of the QR codeword.

4. Blend (mix) the watermarking image and the original image by overlaying the watermarking image on the original image.

Correspondingly, the decoder in Figure 2b decodes the watermarking message (X) from a screen shot of the watermarked image as follows.

5. Detect and extract QR codewords from screenshots.

6. Decode the K-bit binary string from the extracted QR codeword.

7. Decode the watermarking message (X) from the K-bit binary string and side information (Y).

Note that in many implementations, one or more of the above steps (e.g., steps 6-7) can be combined into a single step for better performance.

Note that the QR codeword includes a pattern for detection and an error correction code in a two-dimensional (2D) layout. In some embodiments, a one-dimensional error correction code word may be used in conjunction with a pattern for one-dimensional (1D) detection instead of a QR code word for better performance/flexibility when generating watermarking images. Examples of one-dimensional error correction codes include Reed-Solomon codes, turbo codes, low-density parity check (LDPC) codes, and other general linear block codes.

Considering step 1 of the encoding process above, to determine K, no knowledge of the realization of Y (i.e. the actual side information sequence) is required, but H(X|Y) must be known in advance. Examples of side information (Y) with prior knowledge of H( and information about the Site, including platform information).

The following description uses screenshot timestamps as an example, but other similar implementations could utilize IP address information and/or platform information, or a combination of these.

Recall that the query ID discussed in Figure 1B above is a 128-bit binary string consisting of a timestamp (64 bits), IP address (32 bits), and process id (32 bits) (not including any additional error coding bits). do. In a typical application, the screenshot timestamp (T _s ) is strongly correlated with the timestamp of the query ID (T _q ) such that T _q ≤T _s , which has a high probability of being negative such that T _s -T _q ≤Δ. There is an integer (Δ) that is not.

From this perspective, instead of using 64 bits in this implementation for the timestamp, the encoder in Figure 2b would use K=(ceil(64-log2(Δ)) + 64) bits as an estimate of H(X|Y). , where Y is T _s and “ceil” is a ceiling function that rounds its argument to the nearest integer. As a result, in one embodiment, a binning technique is used to code _Tq , where each bin contains candidate timestamps at least Δ microseconds apart, and the indices of the bins are of length ceil(64-log2( Δ)) is the T _q suffix.

The proposed binning method is based on the fact that the most significant bits are the same for two timestamps that are close to each other. For example, the timestamp for epoch time 2019-01-01 is 1546300800 and its binary is 0b0101 0111 1110 0101 1010 0011 0101 1110 0110 0110 0000 0000 0000.

The timestamp for 2018-01-01 is 1514764800 and its binary version is 0b0101 0110 0001 1010 1011 1010 1001 1101 0010 1000 0000 0000 0000.

In a 64-bit representation, the upper 18 bits are identical. The closer two timestamps are, the more the most significant bits are the same. In a typical implementation, the image timestamp and the screenshot timestamp may be much closer together, typically a day, a week, or a month, so they have more of the same number of bits.

By using the binning scheme described above, in some implementations, the system removes approximately log2(Δ) of the most significant bits from T _q using K = (ceil(64-log2(Δ)) + 64) bits, thereby determining the query ID. can be coded. Figure 1C is an example of a data format 150' for image watermarking according to one such implementation. As shown, the IP address 154 and process ID 156 are the same as in the implementation of Figure 1B, but the timestamp is reduced to some of the least significant bit 158, and additional data 160 reduces the size of the data. It can be added without prompting.

On the decoder side, after extracting and decoding the QR code from the received screenshot to obtain a binary string of K bits, the timestamp LSB 158 can identify the index of the bin containing the exact timestamp (T _q ). . To recover T _q , the decoder can obtain T' _q by combining the first log2(Δ) bit of T _s with the (64-log2(Δ))-bit bin index. If log2(Δ) is not an integer, the smallest integer greater than log2(Δ), i.e. ceil(log2(Δ))), is used instead. In many implementations, with high probability, T _s -T _q ≤Δ, so in the decoder, with high probability, T' _q =T _q . In the rare case that T _s -T _q >Δ and T _s -T _q ≤mΔ (where m is a positive integer), T _q must belong to the following list of size m.

{T' _q , T' _q -Δ, T' _q -2Δ,...,T' _q -(m-1)Δ}.

Since (ceil(64-log2(Δ)) + 64) < 128, these implementations effectively reduce the message rate required to recover the query ID at the decoder. This reduction can be exploited in two ways to improve decoding success rate.

1. Increase the correction level of the selected QR code by including additional parity or error correction bits, or

2. Use a smaller macro QR code (e.g. macro 17).

A 21×21 QR code can store up to 152 bits of information, as listed in the following table.

By reducing the number of bits from 128 to K=(ceil(64-log2(Δ) + 64), the system can use higher Error correction code (ECC) level or smaller QR codes can be used.

Improved message speed

The implementations discussed above use side information available at the decoder to improve the robustness of watermarking. From another perspective, the system may utilize side information to improve message speed.

In this implementation, the encoder of Figure 2b can embed a watermarking message into the image as follows.

1. Convert the 128-bit query ID to a 128-bit binary string with K bits of additional information. Here K is determined by H(X)-H(X|Y).

2. Convert the 128-bit binary string into a QR codeword.

3. Create a watermarked image containing at least one copy of the QR codeword.

4. Overlay the watermarking image on the source image to blend the watermarking image and the source image.

Correspondingly, the decoder in Figure 2b can decode the watermarking message (X) from a screenshot of the image as follows.

4. Detect and extract QR codewords from screenshots.

5. Decode the 128-bit binary string from the extracted QR codeword.

6. Decode the 128-bit query ID along with K bits of additional information from the 128-bit binary string and side information (Y).

Compared to systems that do not implement these methods, these implementations provide additional K-bit messaging capabilities essentially for free, i.e., with the same decoding success rate and same distortion level. These additional K bits can be used to provide better tracking capabilities and/or user experience in terms of ease of use.

As mentioned above, although primarily discussed in terms of reducing data size for timestamps within watermark data, similar implementations could be used with binning applied to IP addresses and/or process identifiers. For example, if the common process identifiers are all less than 20 bits in length, 12 bits may be removed from the process ID 156 (MSB). Likewise, part of the IP address within the watermark data (e.g., the leftmost 8 bits) is derived from side information available to the decoder (e.g., the IP address used to submit the screenshot, the IP address of the decoder, etc.) It can be. To further reduce data size, combinations of these fields can be processed in this way.

Figure 3 is a block diagram of a system for image watermarking according to some implementations. Client device 300, which may include a desktop computer, laptop computer, tablet computer, wearable computer, smart phone, embedded computer, smart car, or any other type and form of computing device, may be connected to one or more devices via network 312. Can communicate with the server 314.

In many implementations, client device 300 may include a processor 302 and a memory device 306. Memory device 306 may store machine instructions (instructions) that, when executed by the processor, cause the processor to perform one or more operations described herein. Processor 302 may include a microprocessor, ASIC, FPGA, or a combination thereof. In many implementations, the processor may be a multi-core processor or processor array. Memory device 306 may include, but is not limited to, an electronic, optical, magnetic, or any other storage device capable of providing program instructions to the processor. The memory device may include a floppy disk, CD-ROM, DVD, magnetic disk, memory chip, ROM, RAM, EEPROM, EPROM, flash memory, optical media, or other suitable memory from which the processor can read instructions. Instructions may contain code in any suitable computer programming language, such as C, C++, C#, Java, JavaScript, Perl, HTML, XML, Python, and Visual Basic.

Client device 300 may include one or more network interfaces 304. Network interface 304 may be any type and form of interface, including 10 Base T, 100 Base T, or 1000 Base T (“Gigabit”) Ethernet; Various 802.11 wireless such as 802.11a, 802.11b, 802.11g, 802.11n or 802.11ac; Cellular, including CDMA, LTE, 3G, or 4G cellular; Bluetooth or other short-range wireless connection; or any combination of these or other interfaces for communicating with a network. In many implementations, client device 300 may include a plurality of network interfaces 304 of different types that allow connectivity to various types of networks 312 . Accordingly, network 312 includes a local area network (LAN), a wide area network (WAN) such as the Internet, a cellular network, a broadband network, a Bluetooth network, an 802.11 (WiFi) network, a satellite network, or a combination of these or other networks. and may include one or more additional devices (e.g., router, switch, firewall, hub, network accelerator, cache, etc.).

A client device may include one or more user interface devices. A user interface device generates sensory information (e.g., a visualization on a display, one or more sounds, tactile feedback, etc.) to convey data to the user or converts sensory information received from the user into electronic signals (e.g., a keyboard, mouse, etc.). It can be any electronic device that converts into a pointing device, touch screen display, microphone, etc.). One or more user interface devices may be internal to the housing of the client device, such as a built-in display, touch screen, microphone, etc., or external to the housing of the client device, such as a monitor connected to the client device, or speakers connected to the client device, depending on various implementations.

Memory 306 may include an application 308 for execution by process 302. Application 308 may include any type and form of application, such as a media application, web browser, productivity application, or any other such application. Application 308 may receive an image including a watermark embedded within the image from a content server and display it through a user interface for a user of the client device.

Memory 306 may also include a capture engine 310, which may be part of an application 308 (e.g., a plug-in or extension of a browser) and/or a part of the device's operating system. Capture engine 310 may include an application, server, service, daemon, routine, or other executable logic for capturing screen shots of rendered images including a watermark. Capture engine 310 may be configured to capture screen shots of all or some images. For example, in some implementations, capture engine 310 may respond to metadata of an image or in response to a script executed by application 308 (e.g., in a script embedded in a web page displayed by a browser). in response) may be triggered to take a screenshot of the image. Capture engine 310 may, in some implementations, take a screenshot of just an image or a screenshot of the entire display or screen. In a further implementation, the capture engine may crop the captured image into a desired image. This can be done for example based on image display coordinates within the display. Capture engine 310 may add metadata, such as capture time (e.g., epoch time) to the screen shot, as described above. Capture engine 310 may also transmit screen shots to a monitoring server via network interface 304. In some implementations, capture engine 310 may include a script that is embedded in a web page and executed by application 308 while rendering the web page, which may also enable the capture engine to capture screenshots. You can include embedded images or image links to help you.

Server 314 may include a content server and/or a monitoring server, which may be the same or a different device. Server(s) 314 may include one or more processors 302, network interface 304, and memory devices 306. Content server 314 may include one or more content items 316 in storage, such as images to be watermarked as well as other content (e.g., web pages, other media, etc.). Content server 314 may also include encoder 202, as discussed above with respect to FIGS. 2A and 2B. Encoder 202 may include software, hardware, or a combination of hardware and software. For example, encoder 202 may include an ASIC, FPGA, or other dedicated hardware for embedding a watermark in an image.

The monitoring server may include a decoder 204 as discussed above with respect to FIG. 2B. Decoder 204 may include software, hardware, or a combination of hardware and software. For example, decoder 204 may include an ASIC, FPGA, or other dedicated hardware for identifying and decoding watermarks from images. As discussed above, decoder 204 may receive additional information helpful in decoding the watermark, such as the screen shot time from metadata of the screen shot received from capture engine 310.

4 is a flow diagram of an image watermarking method according to some implementations. At step 402, the client device may request a content item. The request may be triggered during rendering of a web page by a browser or other application (eg, an interstitial advertising content item at rest in a mobile game or other type and form of content). At step 404, content server 314 may select a content item. Content items may be selected through any means, and may be based on client device type, user account or device identifier, context item within a web page or other application, or any other such information.

At step 406, the content server 314 generates a water tag that may include one or more identifiers, including a timestamp, an identifier of the server or the IP address of the server, and/or a process identifier of the process used to select the content item. A mark identifier can be created. In some implementations, the watermark identifier may include additional information, such as the identifier of the content item. At step 408, the content item may be encoded with a watermark. As discussed above, encoding a content item involves creating an overlay with an alpha channel with pixels modified from the default or a pattern representing the changed bits of the encoded watermark (e.g., a QR code or pseudocode). It can be included. The watermark may repeat at predetermined intervals or intervals throughout the image. The overlay can then be blended or combined with the image to create an encoded content item. At step 410, the encoded content item may be transmitted by the content server to the client device.

Although the content server is shown generating the watermark identifier and encoding the watermark after receiving the content item request, in some implementations, the content item may be pre-encoded (e.g., prior to step 402), The content server may select pre-encoded content items for delivery. In many implementations, this pre-encoding may be performed within a predetermined time frame prior to the request. For example, a content item may be encoded with a given timestamp and used for a predetermined period of time (e.g., two weeks) and then replaced or re-encoded with a new timestamp. This allows the content server to perform encoding processing during less busy times while ensuring that the content and timestamps are relatively up to date. As discussed above, the shorter the period over which a pre-encoded content item can be used, the more data can be encoded into the watermark and/or the more robust the watermark can be. However, even in the example described above, a period of one year or more could be used, significantly reducing the data required.

At step 412, the client device may render a content item within an application, such as a web browser, media player, game, or other application. At step 414, the client device's capture engine (running as a separate service or application's plug-in or script) may capture a screen shot of the content item. Screenshots can be cropped or limited to content items, the entire screen, or a portion of the screen. Screenshots may be identified through metadata that has a capture time stamp and may include other identifiers (e.g., device identifiers, context identifiers of the application and/or web page, etc.). In other implementations, the capture timestamp may be provided through other means. For example, in some implementations, when the capture time and the screenshot transmission time to the server are likely to be very close (e.g., within a few seconds), the packet transmission time (e.g., a timestamp in the transport layer header) (identified or extracted from the packet header, such as the options field) or reception time can be utilized as the capture timestamp. At step 416, the client device may send a screenshot to the monitoring server.

At step 418, a monitoring server, which may be a content server or another device, may receive the screenshot and, in some implementations, extract a timestamp from the screenshot's metadata or identify the time the screenshot was sent or received. there is. The time stamp can be provided to the decoder of the monitoring server as additional information.

At step 420, the decoder may scan the screen shot to extract any identified watermarks. In some implementations, where a watermark appears multiple times in a screenshot, the decoder compares the identified watermarks and determines which watermark has the least distortion (e.g., a watermark that matches most other identified watermarks in the image). You can select or create a watermark that is the average of the marks, etc. At step 422, the decoder may convert the watermark to a string.

At step 424, the decoder may generate a timestamp from a portion of the timestamp extracted at step 418 (e.g., a predetermined number of least significant bits) and use the generated timestamp (e.g. For example, the decoding of a string can be tested (by applying an error correction algorithm to the decoded string with a generated timestamp). If the string is decoded correctly according to the error correction bits, at step 426, the monitoring server may process the screenshot image or data associated with the content item (e.g., identifying the content server via IP address and process identifier, Compare screenshots to original content items to detect rendering distortions or corruption, track delivery of content items, etc.)

If the string is not decoded correctly, the decoder can advance the generated timestamp according to the Δ value from the binning scheme and retest the decoding in step 424. This may be repeated iteratively until decoding is successful, or until all empty index values are tested (suggesting that the watermark has been corrupted, improperly extracted, or a content item prior to the aforementioned usage time window has been utilized). If all empty index values are tested and decoding fails, the decoder may report the error at step 428 to an administrator or user of the system.

Accordingly, the systems and methods discussed herein provide improved image watermarking to improve robustness and capacity without compromising perceptibility. In particular, the systems and methods discussed herein allow for higher decoding success rates at the same distortion level and message rate or higher message rates at the same distortion level and decoding success rate. Implementations of these systems can achieve asymptotic lossless data compression by exploiting an additional chain of additional information that is only available at the decoder and not at the encoder, allowing the same message to be transmitted with fewer bits.

Implementations of the subject matter and operations described herein may be implemented in digital electronic circuitry, or computer software embodied in a tangible medium, firmware or hardware containing the structures disclosed herein and structural equivalents thereof, or a combination of one or more of these. there is. Implementations of the subject matter described herein may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded in a computer storage medium to be executed by or to control the operation of the data processing device. Alternatively or additionally, the program instructions may be an artificially generated radio signal, e.g., a machine-generated electrical, optical, or electromagnetic signal generated to encode information for transmission to an appropriate receiver device for execution by a data processing device. Can be encoded in the signal. A computer storage medium may be or be comprised within a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination of one or more of these. Moreover, although computer storage media are not propagated signals, computer storage media may include a source or destination of computer program instructions encoded with artificially generated radio signals. Computer storage media may also be or be comprised within one or more individual physical components or media (e.g., multiple CDs, disks, or other storage devices). Accordingly, computer storage media can be tangible.

The operations described herein may be implemented as operations performed by a data processing device on data stored in one or more computer-readable storage devices or received from another source.

The term "client" or "server" includes all kinds of devices, devices and machines for processing data, such as programmable processors, computers, systems on a chip, or any number or combination of the foregoing. The devices are FPGAs (field The device may also include special-purpose logic circuitry, such as a programmable gate array) or an application-specific integrated circuit (ASIC). The device may also include code that, in addition to the hardware, creates an execution environment for the computer program in question, such as processor firmware. , a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or a combination of one or more of these. The device and execution environment may include, for example, web services, distributed computing, and grid computing infrastructure. A variety of computing model infrastructures can be implemented.

A computer program (also known as a program, software, software application, script, or code) may be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and may act as a stand-alone program or module. It may be distributed in any form, including as a component, subroutine, object, or other unit suitable for use in a computing environment. Computer programs can, but do not have to, correspond to files in a file system. A program can be a portion of a file that holds other programs or data (for example, one or more scripts stored in a markup language document), a single file dedicated to that program, or a number of coordination files (for example, one or more modules, subprograms). or a file that stores part of the code). A computer program may be distributed to run on a single computer, located at one site, or distributed across multiple sites and interconnected by a communications network.

The processes and logic flows described herein may be performed by one or more programmable processors executing one or more computer programs to perform operations by operating on input data and producing output. Processes and logic flows may also be performed by special purpose logic circuits, such as FPGAs or ASICs, and devices may also be implemented with these.

Processors suitable for executing computer programs include, for example, general-purpose and special-purpose microprocessors, and any one or more processors for any type of digital computer. Typically, a processor receives instructions and data from read-only memory, random access memory, or both. The essential elements of a computer are a processor to perform operations according to instructions and one or more memory devices to store instructions and data. Typically, a computer also includes, or is operably coupled to, one or more mass storage devices for storing data, such as magnetic, magneto-optical disks, or optical disks, transmitting data, or both. However, computers do not need these devices. Additionally, the computer may be embedded in other devices, such as mobile phones, PDAs, mobile audio or video players, gaming consoles, GPS receivers, or portable storage devices (eg, USB flash drives). Devices suitable for storing computer program instructions and data include, for example, semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices); Magnetic disks (e.g., internal hard disks or removable disks); magneto-optical disk; and all forms of non-volatile memory, media and memory devices, including CD-ROM and DVD-ROM disks. Processors and memories can be supplemented or integrated by special-purpose logic circuits.

To provide interaction with a user, implementations of the subject matter described herein may include a display device, such as a CRT (cathode ray tube), plasma, or LCD (liquid crystal display) monitor, for displaying information to the user and input to the computer by the user. It may be implemented on a computer with a keyboard and a pointing device (e.g., a mouse or trackball) capable of providing. Other types of devices may be used to provide interaction with the user. For example, feedback provided to the user may include any form of sensory feedback, such as visual, auditory, or tactile feedback, and input from the user may include acoustic, vocal, or tactile input. It may be received in any form. The computer may also interact with the user by sending and receiving documents to and from the device used by the user, for example, by sending web pages to the web browser of the user's client device in response to requests received from the web browser.

Implementations of the subject matter described herein may include a back-end component (e.g., a data server), a middleware component (e.g., an application server), or a front-end component (e.g., a user-visible component). may be implemented on a computing system (a client computer with a graphical user interface or web browser capable of interacting with implementations of the subject matter described in the specification), or any combination of one or more such back-end, middleware, or front-end components. You can. The components of the system may be interconnected by any form or medium of digital data communication, for example a communication network. Telecommunications networks include local area networks (“LANs”) and wide area networks (“WANs”), inter-network (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks). may be included.

In circumstances where the systems discussed herein may collect or use personal information about a user, the user may collect personal information (e.g., the user's social networks, social actions or activities, the user's preferences, or You may be given the opportunity to control programs or features that may collect information (information about your location) or whether or how you receive content from content servers or other data processing systems that may be more relevant to you. there is. Additionally, certain data may be anonymized in one or more ways before being stored or used so that personally identifiable information is removed when creating parameters. For example, the user's identity may be anonymized so that no personally identifiable information can be determined about the user, and the user's geographic location may be generalized where the location information is obtained so that the user's specific location is not determined (e.g. , city, zip code, or state level). Accordingly, users can control how information about them is collected and used by content servers.

Although this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or what may be claimed, but rather as descriptions of features specific to particular implementations of particular inventions. Certain functionality described herein in the context of separate implementations may also be implemented in combination in a single implementation. Conversely, various features described in the context of a single implementation may also be implemented in multiple implementations individually or in any suitable subcombination. Moreover, although features may be described above, or even initially claimed, as operating in a particular combination, one or more features may in some cases be excluded from the claimed combination, and the claimed combination may be a sub-combination or It may be about a variation of a sub-combination.

Similarly, although operations are shown in the drawings in a particular order, this should not be construed to require that such operations be performed in the specific order or sequential order shown or that all illustrated operations be performed to achieve the desired results. . Multitasking and parallel processing can be advantageous in certain situations. Moreover, the separation of the various system components in the implementations described above should not be understood as requiring such separation in all implementations, and the program components and systems described are typically integrated together into a single software product or packaged into multiple software products. You must understand that it can happen.

Accordingly, specific implementations of the subject matter have been described. Other implementations are within the scope of the following claims. In some cases, the operations recited in the claims can be performed in a different order and still obtain desirable results. Additionally, the processes depicted in the accompanying drawings do not necessarily require the specific order or sequential order shown to achieve desirable results. In certain implementations, multitasking or parallel processing may be utilized.

Claims (27)

A decoder for a water marking system, said decoder comprising:
From a client device, (i) a screenshot of an image displayed by the client device, wherein the image is watermarked with a string associated with the image metadata associated with the image, and (ii) includes screenshot metadata associated with the screenshot of the image. Receive a packet wherein the screenshot metadata contains a timestamp of the screenshot of the image, the image metadata contains the timestamp of the image, and the string contains a subset of the timestamp of the image, and ,
Extract screenshot metadata and strings from packets,
Using part of the screenshot metadata, decode an identifier from a string, wherein the identifier includes image metadata; and
configured to track delivery of a content item associated with an image that is responsive to the decoded identifier;
A decoder for a watermarking system, wherein the decoder is configured to decode an identifier from a string by concatenating a portion of a timestamp of a screenshot of the image with a subset of the timestamp of the image. delete According to paragraph 1,
The decoder is,
A decoder for a watermarking system configured to extract a timestamp of a screenshot of an image from a header of a packet containing the screenshot of the image and screenshot metadata. delete delete According to paragraph 1,
The string is,
A decoder for a watermarking system that includes a number of error correction bits greater than the difference between the length of the timestamp of the image and the length of a subset of the timestamps of the image. According to paragraph 1,
The string includes error correction bits,
The decoder is configured to decode an identifier from a string by iteratively combining a portion of the screenshot metadata with a multiple of a predetermined offset until successfully decoding the identifier, wherein successfully decoding the identifier results in an error correction bit. A decoder for a watermarking system that includes correctly decoding the string according to the decoder. According to paragraph 1,
The decoder is,
A decoder for a watermarking system configured to decode an identifier from a string by repeatedly combining portions of the screenshot metadata with multiples of a predetermined offset until the identifier is successfully decoded. According to paragraph 1,
The string is,
A decoder for a watermarking system that contains the address of the content server that created the watermarked image as a string. According to clause 9,
The string is,
A decoder for a watermarking system that contains the process identifier of the content server that created the watermarked image as a string. As a watermarking method,
From a client device, by a decoder on the device, (i) a screenshot of an image displayed by the client device—the image is watermarked with a string associated with image metadata associated with the image; and (ii) a screen shot associated with the image. Receiving a packet containing shot metadata, wherein the screenshot metadata includes a timestamp of a screenshot of an image, the image metadata includes a timestamp of the image, and the string is a time stamp of the image. Contains a subset of stamps;
extracting screenshot metadata and strings from the packet by a decoder;
decoding, by a decoder, an identifier from the string using part of the screenshot metadata, the identifier comprising image metadata; and
tracking, by the decoder, delivery of the content item associated with the image responsive to the decoded identifier;
Decoding the identifier from the string further comprises concatenating a portion of a timestamp of a screenshot of the image with a subset of the timestamp of the image. delete According to clause 11,
A watermarking method further comprising extracting, by a decoder, a timestamp of a screenshot of an image from a header of a packet containing the screenshot of the image. delete delete According to clause 11,
wherein the string contains a number of error correction bits greater than the difference between the length of the timestamp of the image and the length of a subset of the timestamps of the image. According to clause 11,
The string includes error correction bits,
The step of decoding the identifier from the string is,
further comprising repeatedly combining a portion of the screenshot metadata with multiples of a predetermined offset until successfully decoding the identifier, wherein successfully decoding the identifier includes correctly decoding the string according to the flow correction bits. A watermarking method including the steps of: According to clause 11,
The step of decoding the identifier from the string is,
A watermarking method further comprising iteratively combining portions of the screenshot metadata with multiples of a predetermined offset until the identifier is successfully decoded. According to clause 11,
The string is,
A watermarking method that includes the address of the content server that created the watermarked image as a string. According to clause 19,
The string is,
A watermarking method that includes the process identifier of the content server that created the watermarked image as a string. A computer-readable medium comprising instructions that, when executed by a computing device, cause the computing device to perform the method of any one of claims 11, 13, and 16-20. As a watermarking system,
Includes an encoder of a device, wherein the encoder of the device includes:
Receive an image and image metadata associated with the image, the image metadata including a timestamp of the image, and
generate a string from a subset of image metadata,
Encode a watermark from a string, where the string contains a subset of the timestamp of the image, and
configured to embed a watermark into the image;
The decoder of the device or the second device,
From a client device, (i) a screenshot of an image displayed by the client device—the image is watermarked with a string generated from a subset of the image metadata; and (ii) screenshot metadata associated with the screenshot of the image. receive a packet containing,
From a packet containing a screenshot of an image displayed by a client device and said screenshot metadata, a string and additional screenshot metadata are extracted, wherein said additional screenshot metadata is a timestamp of a screenshot of the image by the client device. Including,
Decode an identifier from a string, using part of the additional screenshot metadata, wherein the identifier includes part of the image metadata; and
Track delivery of content items associated with images that respond to decoded identifiers,
Decoding the identifier from the string further comprises concatenating a portion of a timestamp of a screenshot of the image with a subset of the timestamp of the image. delete According to clause 22,
The encoder of the device is,
A watermarking system configured to generate a string from a predetermined number of least significant bits of image metadata. As a watermarking method,
Receiving, by an encoder of the device, an image and image metadata associated with the image, the image metadata including a timestamp of the image;
generating, by the encoder, a string from a subset of the image metadata, the string comprising a subset of the timestamp of the image;
encoding, by an encoder, a watermark from a string; and
Embedding, by the encoder, a watermark in the image;
The decoder of the device or the second device,
From a client device, (i) a screenshot of an image displayed by the client device—the image is watermarked with a string generated from a subset of the image metadata; and (ii) screenshot metadata associated with the screenshot of the image. receive a packet containing,
From a packet containing a screenshot of an image displayed by a client device and said screenshot metadata, a string and additional screenshot metadata are extracted, wherein said additional screenshot metadata is a timestamp of a screenshot of the image by the client device. Including,
Decode an identifier from a string, using part of the additional screenshot metadata, wherein the identifier includes part of the image metadata; and
Track delivery of content items associated with images that respond to decoded identifiers,
Decoding the identifier from the string further comprises concatenating a portion of a timestamp of a screenshot of the image with a subset of the timestamp of the image. delete According to clause 25,
A watermarking method further comprising generating a string from a predetermined number of least significant bits of image metadata.