KR20210079362A - Improved image watermarking - Google Patents

Abstract

The present disclosure provides systems and methods for improved image watermarking to improve robustness and capacity without compromising perceptibility. In particular, the systems and methods discussed herein may allow for a higher decoding success rate at the same distortion level and message rate or a higher message rate at the same distortion level and decoding success rate. Implementations of such a system achieve asymptotic lossless data compression by utilizing an additional chain of additional information available only to the decoder and not the encoder, allowing the same message to be transmitted with fewer bits.

Description

Improved image watermarking

This application claims the benefit of priority to PCT Application No. PCT/US2019/037959, filed on June 19, 2019, entitled "Improved Image Watermarking", which is incorporated herein by reference in its entirety.

Image watermarking is a technique for embedding visually imperceptible data or messages into an image, and can be classified as non-blind or blind, respectively, depending on whether the original image is required for watermark extraction. Blind watermarking is particularly useful in that embedded data can be recovered without accessing the original previously embedded image.

However, the current implementation of blind image watermarking is not perceptible (eg, whether distortion due to embedding the watermarking message can be detected by a viewer), robustness (eg, the success rate of the decoder in decoding the embedded message) ) and capacity (eg, the amount or speed of data at which data can be embedded in an image). In many implementations, increasing one of these can result in sharp performance degradation in the other.

The systems and methods discussed herein provide improved image watermarking to improve robustness and capacity without compromising perceptibility. In particular, the systems and methods discussed herein allow for a higher decoding success rate at the same distortion level and message rate or a higher message rate at the same distortion level and decoding success rate. Implementations of such systems achieve asymptotic lossless data compression by utilizing side chains (or side channels) of additional information available only to the decoder, not the encoder, so that the same message can be transmitted more robustly or with fewer bits. do.

In one aspect, the present disclosure relates to a system for improved watermarking. The system includes a decoder in the device. The decoder receives a capture of an image including at least one embedded watermark, determines a timestamp of the capture, decodes a binary string from the embedded watermark, and uses a portion of the timestamp of the capture, decode the identifier from the binary string containing the timestamp; and output the decoded identifier.

In some implementations, the timestamp of the capture is identified in the metadata of the capture. In some implementations, the decoder is configured to extract a timestamp of the capture from a header of a packet containing the capture. In some implementations, the binary string of the embedded watermark includes a subset of the timestamp of the image. In a further implementation, the decoder is configured to decode the identifier from the binary string by concatenating a portion of the timestamp of the capture with a subset of the timestamp of the image. In another further implementation, the binary string of the embedded watermark includes a number of error correction bits greater than the difference between the length of the timestamp of the image and the length of the subset of timestamps of the image.

In some implementations, the decoder is configured to decode the identifier from the binary string by combining a portion of a timestamp of the capture with a predetermined offset. In a further implementation, the decoder is configured to decode the identifier from the binary string by repeatedly combining a portion of the timestamp of the capture with a multiple of a predetermined offset until successfully decoding the identifier.

In some implementations, the binary string includes the address of the content server that generated the image including the at least one embedded watermark. In a further implementation, the binary string includes the process identifier of the content server that generated the image including the at least one embedded watermark.

In another aspect, the present disclosure relates to a method for improved watermarking. The method includes receiving, by a decoder of the device, a capture of an image comprising at least one embedded watermark from a client device. The method also includes determining, by the decoder, a timestamp of the capture. The method also includes decoding, by the decoder, the binary string from the embedded watermark. The method also includes decoding, by the decoder, the identifier from the binary string comprising the timestamp of the image, using the portion of the timestamp of the capture. The method also includes outputting, by the decoder, the decoded identifier.

In some implementations, the timestamp of the capture is identified in the metadata of the capture. In some implementations, the method includes extracting, by a decoder, a timestamp of the capture from a header of a packet containing the capture. In some implementations, the binary string of the embedded watermark includes a subset of the timestamp of the image. In a further implementation, the method includes associating the portion of the timestamp of the capture with the subset of the timestamp of the image. In another further implementation, the binary string of the embedded watermark includes a number of error correction bits greater than the difference between the length of the timestamp of the image and the length of the subset of timestamps of the image.

In some implementations, the method includes combining a portion of a timestamp of the capture with a predetermined offset. In a further implementation, the method includes iteratively combining a portion of a timestamp of the capture with a multiple of a predetermined offset until the identifier is successfully decoded.

In some implementations, the binary string includes the address of the content server that generated the image including the at least one embedded watermark. In a further implementation, the binary string includes the identifier of the process of the content server that generated the image including the at least one embedded watermark.

In another aspect, the present disclosure relates to a watermarking system. the system receives the image and metadata associated with the image; generate a binary string from a subset of metadata associated with the image; and an encoder of the device configured to encode the watermark from the binary string and embed the watermark in the image. The device and the decoder of the second device recover the metadata associated with the image from a subset of the metadata associated with the image encoded in the embedded watermark and additional metadata associated with display capture of the image at the third device.

In some implementations, the metadata associated with the image includes a timestamp of the image, and the additional metadata includes a timestamp of a display capture of the image at the third device. In some implementations, the encoder of the device is configured to generate a binary string from a predetermined number of least significant bits of metadata associated with the image.

In another aspect, the present disclosure relates to a watermarking method. The method includes receiving, by an encoder of a device, an image and metadata associated with the image. The method also includes generating, by the encoder, a binary string from a subset of metadata associated with the image. The method also includes encoding, by the encoder, the watermark from the binary string. The method also includes embedding, by the encoder, a watermark in the image. The device or the decoder of the second device recovers the metadata associated with the image from a subset of the metadata associated with the image encoded in the embedded watermark and additional metadata associated with display capture of the image at the third device.

In some implementations, the metadata associated with the image includes a timestamp of the image, and the additional metadata includes a timestamp of a display capture of the image at the third device. In some implementations, the method includes generating a binary string from a predetermined number of least significant bits of metadata associated with the image.

The disclosure also provides a computer program comprising instructions that, when executed by the computing device, cause the computing device to perform any of the methods disclosed herein. The disclosure also provides a computer-readable medium comprising instructions that, when executed by the computing device, cause the computing device to perform any of the methods disclosed herein.

Optional features of one aspect may be combined with any other aspect.

The details of one or more implementations are set forth in the accompanying drawings and the description below. Other features, aspects and advantages of the present disclosure will become apparent from the following description, drawings and claims:
1A is an illustration of an example implementation of image watermarking.
1B is an example of a data format for image watermarking according to an implementation.
1C is an example of a data format for image watermarking according to another implementation. and
2A is a block diagram of a system for image watermarking according to one implementation.
2B is a block diagram of a system for image watermarking according to another implementation.
3 is a block diagram of a system for image watermarking in accordance with some implementations.
4 is a flowchart of a method for image watermarking in accordance with some implementations.
Like reference numbers and designations in the various drawings indicate like elements.

Image watermarking is a technique for embedding (inserting) visually imperceptible data or messages into an image, and may be classified as non-blind or blind, respectively, depending on whether the original image is required for watermark extraction. Blind watermarking is particularly useful in that embedded data can be recovered without accessing the original previously embedded image.

For example, referring briefly to FIG. 1A , an example implementation of image watermarking for image 100 is shown. The small watermark code 102 may include an array of sized pixels and may be placed within the image 100 so that it is not visible to the viewer. As shown, the watermark code 102 may be duplicated throughout the image to provide resistance to local artifacts, or other such distortions due to cropping, compression or other damage. Although shown with only a few pixels for clarity, in many implementations the watermark code may include an area having 64 pixels, 128 pixels, or any other amount. Pixel adjustments to define encoding values can be relatively inconspicuous, unlike simple black and white pixels. For example, in many implementations, the pixels that make up the encoded region may have a color that matches or is similar to surrounding pixels but has an adjusted alpha (transparency) value. For example, the encoding may change a pixel with an alpha value of 0 to an alpha value of 10, 50, 100, 255, or some other such value. In some implementations, the code may be detected by identifying pixels that have an alpha value that differs significantly from the surrounding alpha value. In some implementations, differential encoding may be applied by overlay encoding each bit through encoding a different value by changing the alpha value of the pixels within the overlay.

Any kind of data may be encoded in the watermark 102 . Referring briefly to FIG. 1B , a data format 150 for image watermarking according to one implementation is shown. The data format shown includes 128 bits with a 64 bit timestamp 152 (eg, based on epoch time), an IP address 154 and a process identifier 156 . Data in data format 150 may be referred to herein as a query ID. Many implementations also include error correction bits (not shown) to improve decoding of the watermark. For example, this code may be encoded into a QR code using a Reed-Solomon error correction code contained within the mark.

In one such implementation, data may be encoded into the image by the content server prior to providing the image to the client device along with the content server's IP address and the process identifier of the process that created the image. Then, when the image is received and rendered by the client device, the monitoring process of the client device may capture a screen shot of the image and provide the screen shot to the content server or the monitoring server. For example, a monitoring process on the client device may not have access to the image itself (eg, the image may be stored in a memory location on the client device that the monitoring process does not have access to), but (eg, a frame buffer You can take a screenshot of an image (by reading the image data from it or by capturing the image with a camera). The server can decode the watermark to identify the original creation process and server as well as the time the image was created or marked, and compare the screenshot image with the original image. This allows the system to not only identify different aspects of the image, but also automatically identify distortions or image damage caused by the rendering or encoding process on the image. In implementations where the content server and the monitoring server are different, this may in particular allow the monitoring server to identify a particular content server among a plurality of content servers that provided images to the client device. This can be useful for logging (history), tracking and analysis, and can be much easier than retrieving HTTP logs or similar logs from a client device (which may not have access to the monitoring server).

Watermarking efficiency depends on perceptuality sometimes referred to as “D” (eg, whether distortion introduced by embedding a watermarking message can be perceived by the viewer), and robustness sometimes referred to as “E” (e.g. , the success rate of a decoder in decoding an embedded message) and capacity sometimes referred to as “R” (eg, the amount or rate of data at which data can be embedded in an image). In many implementations, it may be desirable to have low perceptibility, high robustness, and high capacity. However, in many implementations, improving one of these may result in a sharp performance penalty in the other implementation. For example, if you want to add more data to a message while maintaining robustness, you need to enlarge the size of the watermark so that it can be more perceptible. Similarly, the mark size can be maintained while adding data by removing the error correction bits, but this of course makes the mark more difficult to decode and more fragile.

2A is a block diagram of a system 200 for image watermarking according to one implementation. The system may include an encoder 202 and a decoder 204 that may be on the same or different computing devices (eg, a content server and a monitoring server). The image ("S") 206 may be encoded by the encoder 202 as a message ("X") 208 to produce a watermark image ("S"') 210 comprising S+X. have. The encoded or watermarked image S′ may be transmitted via communication channel 212 to, for example, a client device. A corresponding watermarked image (eg, from a screen shot as discussed above) may be provided to the decoder 204 . For example, the client device may send the watermarked image to the decoder 204 via the communication channel 212 . Thus, a communication channel can include any combination of networks and devices between encoders and decoders, which can potentially introduce additional distortions of any kind. For example, channels may be lost as a result of intentional or unintentional attack or damage. Examples of unintentional corruption include image rotation, resizing (scaling) and format conversion. Examples of intentional damage include noise injection (eg, adding information), and attempts to remove watermarking codes (eg, subtracting information).

The decoder 204 may detect and decode the watermark from the watermark image (S') to recover the original message (X) 208', and apply error correction as needed (and potentially multiple in the image). ) and compares each decoded message to rule out errors or distortions in a single watermark).

Thus, the encoder encodes a message such as the timestamp/address/process ID string discussed above along with any error correction code into a mark such as a QR code, and at least one copy of the mark through blending of an alpha channel overlay. can be encoded into the image, the decoder can decode the message by detecting inconsistencies that identify overlay patterns and QR codes, decode the original string, and identify the original timestamp/address/process ID. .

Although these systems are relatively successful, they have a high error rate. In the experiment of decoding the embedded mark from the screenshot of the encoded image, the decoding success rate was 44.03%.

As mentioned above, given a fixed message rate (eg 128 bits), the factors affecting the decoding success rate are the distortion introduced into the image by the encoder (D _e ) and the captured screenshot and the encoder. Distortion between watermarked images at the output D _{c .} In general, the robustness of an image watermarked measured in the decoding success rate is controlled by D _e, D _e when it increases could lead to higher decoding success rate for the same D _c. However, in most cases the watermark should not be visually perceptible in the watermarked image. This requirement imposes an upper limit (upper bound) for the D _e. This constraint on D _e essentially implies an upper bound on the decoding success rate for any given channel. _{In some extreme cases where the D c} introduced by the channel is large, the decoding success rate may drop to almost zero, thus limiting the applicability of this watermarking implementation.

(D ₀ , E ₀ , R ₀ ) represent the distortion, decoding success rate and message rate of the implementation of the watermarking method discussed above, respectively. In a typical implementation, an improvement in one of the three quantities necessarily results in a loss of performance in at least one of the other quantities. For example, in order to improve the E _0, to reduce the R ₀ while keeping the expense of the R ₀ D ₀ D ₀ or maintain. However, in many applications, _{both D 0} and R ₀ currently have strict constraints, where D ₀ must be set as an upper bound so as not to negatively affect the user experience, and R ₀ must be set as an upper bound so that the watermarking message is useful for tracking purposes. It should be specified as the lower limit. In this context, the current implementation of watermarking has little room to improve _{E 0 .}

The systems and methods discussed herein provide improved image watermarking to improve robustness and capacity without compromising perceptibility. In particular, the systems and methods discussed herein enable higher decoding success rates at the same distortion level and message rate or higher message rates at the same distortion level and decoding success rate. Implementations of such a system achieve asymptotic lossless data compression by utilizing an additional chain of additional information available only to the decoder, not the encoder, so that the same message can be transmitted with fewer bits.

Because the distortion constraint is imposed by the application, the systems discussed herein focus on the trade-off between decoding success rate and message rate. In particular, the system avoids the lower limit of the message rate mentioned above without compromising the usefulness of the watermarking message. As a result, this provides greater flexibility in finding the right trade-off between robustness and capacity that was not possible in previous implementations. In particular, side information available only at the decoder can be used to achieve asymptotic lossless compression.

2B is a block diagram of a system 200' for image watermarking according to one such implementation. As discussed with respect to FIG. 2A , the encoder 202 sends the image 206 in the message 208 to generate a watermarked image 210 that can be provided to the decoder 204 via the communication channel 212 . ) is encoded. However, to recover the original message 208', the decoder uses additional side information ("Y") 214 that is not available to the encoder. This eliminates the need for separate communication between the encoder and decoder, and can be particularly advantageous in implementations where the content server and monitoring server are not on the same device (which may not be controlled by the same entity).

2A and 2B, the main difference is the introduction of side information Y in the decoder of FIG. 2B. From the classical source coding theorem, the minimum speed required for lossless recovery of message X in the decoder in Fig. 2a is given by the marginal entropy H(X) of X. Correspondingly, from the SlepianWolf coding theorem, the minimum speed required for lossless recovery of message X in the decoder in Fig. 2b is given by the conditional entropy H(X|Y) of X given Y. Since H(X|Y)≤H(X) for any (X,Y), by using side information (Y), a lower message to send the same message (X) in FIG. 2B than in FIG. 2A It is possible to use speed. The stronger the correlation between X and Y, the lower the message rate can be.

improved robustness

In a first implementation, the system may utilize side information Y in the decoder to improve robustness. In some such implementations, the encoder of Figure 2b embeds a watermarking message in the image as follows:

1. Convert the watermarking message (X) to a K-bit binary string, where K is determined by H(X|Y).

2. Convert the K-bit binary string into a QR codeword.

3. Create a watermarked image comprising at least one copy of the QR codeword.

4. Blend (mix) the watermarking image and the original image by overlaying the watermarking image on the original image.

Correspondingly, the decoder of Fig. 2b decodes the watermarking message X from the screen shot of the watermarked image as follows.

5. Detect and extract QR codewords from screenshots.

6. Decode the K-bit binary string from the extracted QR codeword.

7. Decode the watermarking message (X) from the K-bit binary string and side information (Y).

It is noted that in many implementations, one or more of the above steps (eg, steps 6-7) may be combined into a single step for better performance.

Note that a QR codeword contains a pattern for detection and an error correction code in a two-dimensional (2D) layout. In some embodiments, a one-dimensional error correction code word with a pattern for one-dimensional (1D) detection may be used instead of a QR codeword for better performance/flexibility when generating a watermarking image. Examples of one-dimensional error correction codes include Reed-Solomon codes, turbo codes, low density parity check (LDPC) codes, and other general linear block codes.

Considering step 1 of the above encoding process, in order to determine K, knowledge of the realization of Y (that is, the actual side information sequence) is not required, but H(X|Y) must be known in advance. Examples of side information Y with prior knowledge of H(X|Y) include screenshot timestamp, decoding time, and any relevant information about the screenshot (e.g., IP address and geographic location, publisher information). and information about the site including platform information).

Although the following description uses the screenshot timestamp as an example, other similar implementations may utilize IP address information and/or platform information, or a combination thereof.

Recall that the query ID discussed in Figure 1b above is a 128-bit binary string consisting of a timestamp (64 bits), an IP address (32 bits) and a process id (32 bits) (not including any additional error coding bits). do. In a typical application, screenshots timestamp (T _s) is strongly correlated with the time stamp (T _q) of the query ID to be T _q ≤T _s, where there is a high probability of negative, such as T _s -T _q ≤Δ There is an integer (Δ) that is not

In this regard, instead of using 64 bits in this implementation for the timestamp, the encoder of Figure 2b would use K=(ceil(64-log2(Δ)) + 64) bits as an estimate of H(X|Y). where Y is T _s and "ceil" is the zenith function rounding the argument to the nearest integer. Consequently, in one embodiment, a binning technique is _{used to code T q} , where each bin contains a candidate timestamp at least Δ microseconds apart, and the index of the bin is of length ceil(64-log2( Δ)) is the T _q suffix.

The proposed binning scheme is based on the fact that the most significant bit is the same for two time stamps that are close to each other. For example, the timestamp for 2019-01-01 in epoch time is 1546300800 and its binary is 0b0101 0111 1110 0101 1010 0011 0101 1110 0110 0110 0000 0000 0000.

The timestamp for 2018-01-01 is 1514764800 and its binary version is 0b0101 0110 0001 1010 1011 1010 1001 1101 0010 1000 0000 0000 0000.

In a 64-bit representation, the upper 18 bits are identical. The closer the two timestamps are, the more the most significant bits are identical. In a typical implementation, image timestamps and screenshot timestamps can typically be much closer, such as a day, a week, or a month, so they have the same number of bits.

By using the binning scheme described above, in some implementations, the system removes about log2(Δ) of the most significant bit from _{T q} using K = (ceil(64-log2(Δ)) + 64) bits to query ID can be coded. 1C is an illustration of a data format 150' for image watermarking according to one such implementation. As shown, the IP address 154 and process ID 156 are the same as in the implementation of FIG. 1B , but the timestamp is reduced to some of the least significant bits 158 and the additional data 160 reduces the size of the data. It can be added without

On the decoder side, after extracting and decoding the QR code from the received screen shot to obtain a K-bit binary string, the timestamp LSB 158 can identify the index of the bin containing the _{correct timestamp T q .} . To recover T _q , the decoder can obtain _{T' q by combining the first log2(Δ) bits of T s} and the empty index of (64-log2(Δ)) bits. If log2(Δ) is not an integer, the smallest integer greater than log2(Δ), ceil(log2(Δ))), is used instead. _{Since T s} -T _q ≤Δ with high probability in many implementations, _{T' q} =T _q with high probability in the decoder. In the _{rare case where T s} -T _q >Δ, if T _s -T _q ≤ mΔ, where m is a positive integer, then T _q must belong to the following list of sizes m:

{T' _q , T' _q -Δ, T' _q -2Δ,...,T' _q -(m-1)Δ}.

Since (ceil(64-log2(Δ)) + 64)<128, these implementations effectively reduce the message rate required to recover the query ID at the decoder. This reduction can be exploited in two ways to improve the decoding success rate.

1. increase the correction level of the selected QR code by including additional parity or error correction bits; or

2. Use a smaller macro QR code (eg macro 17).

A 21×21 QR code can store up to 152 bits of information as listed in the following table.

By reducing the number of bits from 128 to K=(ceil(64-log2(Δ) + 64), the system uses higher (e.g., changing from middle to quartile) to improve the decoding success rate. Error Correcting Code (ECC) levels or smaller QR codes can be used.

Improved message rate

The implementations discussed above use side information available at the decoder to improve the robustness of the watermarking. In another aspect, the system may utilize side information to improve message rate.

In this implementation, the encoder of FIG. 2B may embed the watermarking message into the image as follows.

1. Convert a 128-bit query ID to a 128-bit binary string with K bits of additional information. where K is determined by H(X)-H(X|Y).

2. Converts a 128-bit binary string into a QR codeword.

3. Create a watermarked image comprising at least one copy of the QR codeword.

4. Blend the watermarking image and the source image by overlaying the watermarking image on the source image.

Correspondingly, the decoder of FIG. 2B may decode the watermarking message X from the screen shot of the image as follows.

4. Detect and extract QR codewords from screenshots.

5. Decode a 128-bit binary string from the extracted QR codeword.

6. Decode 128-bit query ID with K-bit additional information from 128-bit binary string and side information (Y).

Compared to a system that does not implement this method, this implementation provides additional K-bit messaging capabilities for essentially free, i.e. with the same decoding success rate and the same distortion level. These additional K bits can be used to provide better tracking and/or user experience in terms of ease of use.

As mentioned above, although discussed primarily in terms of reducing the data size for timestamps in watermark data, similar implementations may be used with binning applied to IP addresses and/or process identifiers. For example, if the length of a typical process identifier is all less than 20 bits, 12 bits may be removed from the process ID 156 (MSB). Similarly, part of the IP address in the watermark data (eg, the leftmost 8 bits) is derived from additional information available to the decoder (eg, the IP address used to submit the screenshot, the IP address of the decoder, etc.) can be To further reduce the data size, combinations of these fields can be processed in this way.

3 is a block diagram of a system for image watermarking in accordance with some implementations. Client device 300 , which may include a desktop computer, laptop computer, tablet computer, wearable computer, smart phone, embedded computer, smart car, or any other type and form of computing device, may be connected via network 312 to one or more It can communicate with the server 314 .

In many implementations, the client device 300 may include a processor 302 and a memory device 306 . Memory device 306 may store machine instructions (instructions) that, when executed by a processor, cause the processor to perform one or more operations described herein. The processor 302 may include a microprocessor, an ASIC, an FPGA, or a combination thereof. In many implementations, the processor may be a multi-core processor or processor array. Memory device 306 may include, but is not limited to, electronic, optical, magnetic, or any other storage device capable of providing program instructions to a processor. The memory device may include a floppy disk, CD-ROM, DVD, magnetic disk, memory chip, ROM, RAM, EEPROM, EPROM, flash memory, optical medium, or other suitable memory in which the processor can read instructions. The instructions may include code in a suitable computer programming language such as C, C++, C#, Java, JavaScript, Perl, HTML, XML, Python and Visual Basic.

The client device 300 may include one or more network interfaces 304 . The network interface 304 may be a 10 base T, 100 base T, or 1000 base T (“Gigabit”) interface of any type and form, including Ethernet; various 802.11 radios such as 802.11a, 802.11b, 802.11g, 802.11n or 802.11ac; cellular, including CDMA, LTE, 3G or 4G cellular; Bluetooth or other short-range wireless connection; or any combination of these or other interfaces for communicating with the network. In many implementations, the client device 300 may include a plurality of network interfaces 304 of different types that allow connections to various types of networks 312 . Accordingly, network 312 may include a local area network (LAN), a wide area network (WAN) such as the Internet, a cellular network, a broadband network, a Bluetooth network, an 802.11 (WiFi) network, a satellite network, or a combination of these or other networks. and may include one or more additional devices (eg, routers, switches, firewalls, hubs, network accelerators, caches, etc.).

A client device may include one or more user interface devices. The user interface device generates sensory information (eg, a visualization on a display, one or more sounds, tactile feedback, etc.) to transmit data to the user or transmit sensory information received from the user to an electronic signal (eg, keyboard, mouse, pointing device, touch screen display, microphone, etc.). One or more user interface devices may be within the housing of the client device, such as an embedded display, touch screen, microphone, etc., or may be external to the housing of the client device, such as a monitor connected to the client device, speakers connected to the client device, according to various implementations.

Memory 306 may contain applications 308 for execution by process 302 . Applications 308 may include any type and type of application, such as a media application, a web browser, a productive application, or any other such application. The application 308 may receive an image including a watermark embedded in the image from a content server and display it via a user interface for a user of the client device.

The memory 306 may also include a capture engine 310 which may be part of the application 308 (eg, a plug-in or extension of a browser) and/or part of the device's operating system. The capture engine 310 may include an application, server, service, daemon, routine, or other executable logic for capturing a screen shot of a rendered image including a watermark. The capture engine 310 may be configured to capture screen shots of all images or some images. For example, in some implementations, the capture engine 310 responds to metadata of an image or in response to a script executed by the application 308 (eg, to a script embedded in a web page displayed by a browser). in response) to take a screenshot of the image. Capture engine 310 may, in some implementations, just take a screenshot of an image or take a screenshot of the entire display or screen. In a further implementation, the capture engine may crop the captured image to a desired image. This can be done, for example, based on the image display coordinates in the display. The capture engine 310 may add metadata such as capture time (eg, epoch time) to the screen shot as described above. The capture engine 310 may also send the screenshot to the monitoring server via the network interface 304 . In some implementations, the capture engine 310 may include scripts embedded in web pages and executed by the application 308 while rendering the web pages, which web pages also enable the capture engine to capture screenshots. You can include an embedded image or image link to help you.

Server 314 may include a content server and/or monitoring server, which may be the same or a different device. The server(s) 314 may include one or more processors 302 , a network interface 304 , and a memory device 306 . The content server 314 may include one or more content items 316 in storage, such as images to be watermarked as well as other content (eg, web pages, other media, etc.). The content server 314 may also include an encoder 202 as discussed above with respect to FIGS. 2A and 2B . The encoder 202 may include software, hardware, or a combination of hardware and software. For example, the encoder 202 may include an ASIC, FPGA, or other dedicated hardware for embedding a watermark in an image.

The monitoring server may include a decoder 204 as discussed above with respect to FIG. 2B . The decoder 204 may include software, hardware, or a combination of hardware and software. For example, the decoder 204 may include an ASIC, FPGA, or other dedicated hardware for identifying and decoding a watermark from an image. As discussed above, the decoder 204 may receive additional information from the capture engine 310 to aid in watermark decoding, such as the screen shot time from metadata of the received screen shot.

4 is a flowchart of an image watermarking method in accordance with some implementations. At step 402 , the client device may request a content item. The request may be triggered during rendering of a web page by a browser or other application (eg, an interstitial content item resting in a mobile game or other type and form of content). At step 404, the content server 314 may select a content item. The content item may be selected through any means, and may be based on a client device type, a user account or device identifier, a context item in a web page or other application, or any other such information.

In step 406, the content server 314 sends water that may include one or more identifiers including a timestamp, the server's identifier or IP address of the server, and/or the process identifier of the process used to select the content item. You can create a mark identifier. In some implementations, the watermark identifier may include additional information, such as an identifier of a content item. At step 408, the content item may be encoded with a watermark. As discussed above, encoding a content item involves creating an overlay with an alpha channel with pixels modified from a default or pattern representing a changed bit (eg, a QR code or similar code) of the encoded watermark. may include The watermark may be repeated at predetermined intervals or intervals across the image. The overlay can then be blended or combined with the image to create an encoded content item. In step 410, the encoded content item may be transmitted by the content server to the client device.

Although the content server is shown to generate the watermark identifier and encode the watermark after receiving the content item request, in some implementations the content item may be pre-encoded (eg, prior to step 402 ); The content server may select the pre-encoded content item for delivery. In many implementations, this pre-encoding may be performed within a predetermined time frame prior to the request. For example, a content item may be encoded with a given timestamp and used for a predetermined period of time (eg, two weeks) and then replaced or re-encoded with a new timestamp. This allows the content server to perform encoding processing during less busy times while ensuring that the content and timestamps are relatively up-to-date. As discussed above, the shorter the period over which a pre-encoded content item can be used, the more data can be encoded in the watermark and/or the more robust the watermark can be. However, even in the example described above, a period of one year or more can be used while greatly reducing the data required.

At step 412 , the client device may render the content item within an application such as, for example, a web browser, media player, game, or other application. At step 414 , the client device's capture engine (executed as a plug-in or script in a separate service or application) may capture a screen shot of the content item. Screenshots may be cropped, limited to content items, or may be the entire screen or part of the screen. Screenshots may be identified via metadata with capture timestamps and may include other identifiers (eg, device identifiers, context identifiers of applications and/or web pages, etc.). In other implementations, the capture timestamp may be provided through other means. For example, in some implementations, when the capture time and the screen shot transmission time to the server are likely to be very close (eg within a few seconds), the packet transmission time (eg, a timestamp in the transport layer header) (identified or extracted from the packet header, such as an option field) or the reception time may be utilized as the capture timestamp. In step 416 , the client device may send the screen shot to the monitoring server.

At step 418, a monitoring server, which may be a content server or other device, may receive the screenshot, and in some implementations may extract a timestamp from the metadata of the screenshot or identify the time the screenshot was sent or received. have. The time stamp may be provided to the decoder of the monitoring server as additional information.

In step 420, the decoder may scan the screen shot to extract any identified watermarks. In some implementations where the watermark appears multiple times in the screen shot, the decoder compares the identified watermarks and the watermark with the least distortion (e.g., the matching watermark of the most other watermarks in the image, the other watermark identified A watermark that is an average of the marks, etc.) or create In step 422, the decoder may convert the watermark to a string.

At step 424, the decoder may generate a timestamp from a portion of the timestamp extracted in step 418 (eg, a predetermined number of least significant bits) and using the generated timestamp (e.g., For example, by applying an error correction algorithm to the decoded character string having the generated time stamp), the decoding of the character string may be tested. If the string is decoded correctly according to the error correction bits, at step 426, the monitoring server may process the screenshot image or data associated with the content item (eg, identify the content server via IP address and process identifier; Compare screenshots to original content items, track delivery of content items, etc.) to detect rendering distortion or corruption.

If the character string is not decoded correctly, the decoder may advance the generated timestamp according to the Δ value from the binning scheme, and may retest decoding in step 424 . This may be iteratively repeated until either the decoding succeeds, or all bin index values have been tested (suggesting that the watermark has been corrupted or improperly extracted, or that a content item prior to the aforementioned usage time window has been utilized). If all bin index values have been tested and decoding fails, then in step 428 the decoder may report an error to an administrator or user of the system.

Accordingly, the systems and methods discussed herein provide improved image watermarking to improve robustness and capacity without compromising perceptibility. In particular, the systems and methods discussed herein allow for higher decoding success rates at the same distortion level and message rate or higher message rates at the same distortion level and decoding success rate. Implementations of such a system can achieve asymptotic lossless data compression by utilizing an additional chain of additional information available only to the decoder, not the encoder, so that the same message is transmitted with fewer bits.

Implementations of the subject matter and operations described herein may be implemented in digital electronic circuitry, or as computer software embodied in firmware or hardware, or a combination of one or more of them, in a tangible medium comprising the structures disclosed herein and structural equivalents thereof. have. Implementations of the subject matter described herein may be implemented as one or more computer programs, ie, one or more modules of computer program instructions executed by a data processing device or encoded in a computer storage medium for controlling the operation of the data processing device. Alternatively or additionally, the program instructions may be an artificially generated radio signal, eg, a machine generated electrical, optical or electromagnetic generated to encode information for transmission to a receiver device suitable for execution by a data processing device. may be encoded in the signal. A computer storage medium may be or be included in a computer readable storage device, a computer readable storage substrate, a random or serial access memory array or device, or a combination of one or more thereof. Moreover, while computer storage media is not a propagated signal, computer storage media may include a source or destination of computer program instructions encoded in an artificially generated propagated signal. A computer storage medium may also be or be included in one or more separate physical components or media (eg, multiple CDs, disks, or other storage devices). Accordingly, computer storage media can be tangible.

The operations described herein may be implemented as operations performed by a data processing apparatus on data stored in one or more computer-readable storage devices or received from other sources.

The term "client" or "server" includes all kinds of apparatus, devices and machines for processing data, such as programmable processors, computers, systems on a chip, or many or combinations of the foregoing. programmable gate arrays) or special purpose logic circuits such as ASICs (application specific integrated circuits) A device may also include, in addition to hardware, code that creates an execution environment for its computer program, for example, processor firmware. , protocol stacks, database management systems, operating systems, cross-platform runtime environments, virtual machines, or a combination of one or more of these devices and execution environments, such as web services, distributed computing and grid computing infrastructure. Various computing model infrastructures can be implemented.

A computer program (also called a program, software, software application, script or code) may be written in any form of programming language, including compiled or interpreted language, declarative or procedural language, and may be a stand-alone program or module; It may be distributed in any form, including components, subroutines, objects, or other units suitable for use in a computing environment. A computer program can, but need not, correspond to a file in a file system. A program may be a part of another program or file holding data (e.g., one or more scripts stored in a markup language document), a single file dedicated to that program, or multiple control files (e.g., one or more modules, subprograms). or in a file that stores portions of code). The computer program may be distributed to be executed on one computer or multiple computers located at one site or distributed over multiple sites and interconnected by a communication network.

The processes and logic flows described herein may be performed by one or more programmable processors executing one or more computer programs to perform operations by operating on input data and generating output. Processes and logic flows may also be performed by special purpose logic circuits, such as FPGAs or ASICs, and devices may also be implemented with them.

Processors suitable for the execution of a computer program include, for example, general and special purpose microprocessors, and any one or more processors for digital computers of any kind. Processors typically receive instructions and data from read-only memory, random access memory, or both. Essential elements of a computer are a processor for performing operations according to instructions and one or more memory devices for storing instructions and data. In general, a computer also includes, or is operatively coupled to, one or more mass storage devices for storing data, for example, receiving data from, sending data to, or receiving data from, for example, a magnetic, magneto-optical disk, or optical disk. However, a computer does not need such a device. The computer may also be embedded in another device, such as a mobile phone, PDA, mobile audio or video player, game console, GPS receiver, or portable storage device (eg, a USB flash drive). Devices suitable for storing computer program instructions and data include, for example, semiconductor memory devices (eg, EPROM, EEPROM, and flash memory devices); magnetic disks (eg, internal hard disks or removable disks); magneto-optical disk; and all forms of non-volatile memory, media and memory devices, including CD-ROM and DVD-ROM disks. The processor and memory may be supplemented or integrated by special purpose logic circuitry.

To provide for interaction with a user, implementations of the subject matter described herein may include a display device such as a CRT (cathode ray tube), plasma or LCD (liquid crystal display) monitor for displaying information to the user and input by the user into the computer. may be implemented in a computer with a keyboard and pointing device (eg, mouse or trackball) that may provide Other types of devices may be used to provide interaction with the user. For example, the feedback provided to the user may include any form of sensory feedback, such as visual feedback, auditory feedback or tactile feedback, and input from the user may include acoustic, voice, or tactile input. It may be received in any form. In addition, the computer may interact with the user by exchanging documents with the device used by the user, for example, by sending a web page to the web browser of the user's client device in response to a request received from the web browser.

Implementations of the subject matter described herein may include a back-end component (eg, a data server), a middleware component (eg, an application server), or a front-end component (eg, a user to be implemented on a computing system comprising any combination of one or more such backend, middleware or frontend components) or a client computer having a web browser or graphical user interface capable of interacting with implementations of the subject matter described in the specification. can The components of the system may be interconnected by any form or medium of digital data communication, for example, a communication network. Telecommunication networks include local area networks (“LANs”) and wide area networks (“WANs”), network-to-network (eg, the Internet), and peer-to-peer networks (eg, ad hoc peer-to-peer networks). may be included.

In the circumstances in which the systems discussed herein may collect or use personal information about a user, the user may use personal information (e.g., the user's social networks, social actions or activities, the user's preferences, or the user's You may be provided with the opportunity to control programs or features that may collect information about your location) or to control whether or how we receive content from content servers or other data processing systems that may be more relevant to you. have. Additionally, certain data may be anonymized in one or more ways before being stored or used, so personally identifiable information may be removed when creating parameters. For example, the user's identity may be anonymized so that personally identifiable information cannot be determined about the user, and the user's geographic location is generalized where location information is obtained so that the user's specific location is not determined (e.g. , city, zip code or state level). Thus, users can control how information about themselves is collected and used by the content server.

While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or of what may be claimed, but rather as descriptions of features specific to particular implementations of particular inventions. Certain features that are described herein in the context of separate implementations may be implemented in combination in a single implementation. Conversely, various functions that are described in the context of a single implementation may be implemented in multiple implementations individually or in any suitable subcombination. Moreover, although features may be described above and even initially claimed as acting in a particular combination, one or more features from a claimed combination may in some cases be excluded from the combination, and the claimed combination may be a sub-combination or It may be for a variant of a sub-combination.

Similarly, while acts are shown in the figures in a particular order, this should not be construed as requiring that such acts be performed in the particular order shown or sequential order or that all illustrated acts be performed to achieve desirable results. . Multitasking and parallel processing can be advantageous in certain situations. Moreover, the separation of various system components in the implementations described above should not be understood as requiring such separation in all implementations, and the described program components and systems are typically integrated together into a single software product or packaged into multiple software products. You have to understand that it can be.

Accordingly, specific implementations of the subject matter have been described. Other implementations are within the scope of the following claims. In some cases, the actions recited in the claims may be performed in a different order and still achieve desirable results. Moreover, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or sequential order to achieve desirable results. In certain implementations, multitasking or parallel processing may be used.

Claims (27)

A decoder for a watermarking system, the decoder comprising:
From the client device, (i) a screen shot of an image displayed by the client device, wherein the image is watermarked with a string associated with metadata associated with the image and (ii) a packet containing metadata associated with the screen shot of the image receive,
extract from the packet, string and image related metadata related to the screenshot,
decode an identifier from the string using a part of metadata related to the screenshot of the image, wherein the identifier includes metadata related to the image sheet; And
A decoder for a watermarking system configured to track delivery of a content item associated with an image in response to the decoded identifier. According to claim 1,
The metadata associated with the screenshot of the image includes a timestamp of the screenshot of the image, and the decoded identifier includes a timestamp of the image. 3. The method of claim 2,
The decoder is
A decoder for a watermarking system, configured to extract a timestamp of a screenshot of an image from a header of a packet comprising a screenshot of the image and metadata associated with the screenshot of the image. 4. The method of claim 2 or 3,
wherein the string includes a subset of timestamps of the image. 5. The method of claim 4,
The decoder is
A decoder for a watermarking system, configured to decode an identifier from a string by concatenating a portion of a timestamp of a screen shot of the image with a subset of the timestamp of the image. 6. The method according to claim 4 or 5,
The string is
A decoder for a watermarking system comprising a plurality of error correction bits greater than a difference between a length of a timestamp of an image and a length of a subset of timestamps of the image. According to any one of the preceding claims,
The decoder is
A decoder for a watermarking system, configured to decode an identifier from a string by combining a portion of metadata associated with a screenshot of an image with a predetermined offset. 8. The method of claim 7,
The decoder is
A decoder for a watermarking system, configured to decode an identifier from a string by repeatedly combining a portion of metadata associated with a screenshot of an image with a multiple of a predetermined offset until successfully decoding the identifier. According to any one of the preceding claims,
The string is
A decoder for a watermarking system that contains the address of the content server that generated the watermarked image as a string. 10. The method of claim 9,
The string is
A decoder for a watermarking system that includes the process identifier of the content server that generated the watermarked image as a string. A watermarking method comprising:
by the device's decoder from the client device, (i) a screen shot of an image displayed by the client device, wherein the image is watermarked with a string associated with metadata associated with the image and (ii) meta data associated with the screen shot of the image receiving a packet comprising data;
extracting, by the decoder, from the packet, metadata related to the character string and the screenshot of the image;
decoding, by a decoder, an identifier from the string, using a portion of metadata related to the screenshot of the image, the identifier including metadata related to the image sheet; and
and tracking, by a decoder, delivery of a content item associated with the image in response to the decoded identifier. 12. The method of claim 11,
The metadata associated with the screenshot of the image includes a timestamp of the screenshot of the image, and the decoded identifier includes a timestamp of the image. 13. The method of claim 12,
and extracting, by a decoder, a timestamp of the screen shot of the image from the header of the packet containing the screenshot of the image. 14. The method of claim 12 or 13,
wherein the string includes a subset of the timestamp of the image. 15. The method of claim 14,
Decoding the identifier from the string comprises:
and associating a portion of the timestamp of the screenshot of the image with a subset of the timestamp of the image. 16. The method of claim 14 or 15,
wherein the string includes a plurality of error correction bits greater than a difference between the length of the timestamp of the image and the length of the subset of timestamps of the image. 17. The method according to any one of claims 11 to 16,
Decoding the identifier from the string comprises:
and associating a portion of the metadata associated with the screenshot of the image with a predetermined offset. 18. The method of claim 17,
Decoding the identifier from the string comprises:
and iteratively combining a portion of the metadata associated with the screenshot of the image with a multiple of a predetermined offset until the identifier is successfully decoded. 19. The method according to any one of claims 11 to 18,
The string is
A watermarking method including the address of the content server that generated the watermarked image as a string. 20. The method of claim 19,
The string is
A watermarking method including the process identifier of the content server that generated the watermarked image as a string. A computer-readable medium comprising instructions that, when executed by a computing device, cause the computing device to perform the method of any one of claims 11-20. A water marking system comprising:
an encoder of a device, wherein the encoder of the device comprises:
receive images and metadata associated with them;
create a string from a subset of metadata associated with the image,
encode the watermark from the string, and
and embed a watermark into the image;
The device or the decoder of the second device,
From the client device, (i) a screen shot of an image displayed by the client device, wherein the image is watermarked with a string generated from a subset of metadata associated with the image and (ii) metadata associated with the screen shot of the image. receive a packet containing
extracting, from a packet containing a screenshot of the image displayed by the client device and metadata related to the screenshot of the image, a string and additional metadata related to the screenshot of the image;
decode an identifier from the string using a portion of additional metadata associated with the screenshot of the image, wherein the identifier includes a portion of metadata associated with the image; And
A watermarking system that tracks delivery of content items associated with images in response to decoded identifiers. 23. The method of claim 22,
The metadata associated with the image includes a timestamp of the image, and the additional metadata includes a timestamp of a screen shot of the image by a client device. 24. The method of claim 22 or 23,
The encoder of the device,
A watermarking system configured to generate a string from a predetermined number of least significant bits of metadata associated with an image. A watermarking method comprising:
receiving, by an encoder of the device, an image and metadata associated with the image;
generating, by the encoder, a string from a subset of metadata associated with the image;
encoding, by the encoder, the watermark from the character string; and
embedding, by the encoder, a watermark in the image;
The device or the decoder of the second device,
From the client device, (i) a screen shot of an image displayed by the client device, wherein the image is watermarked with a string generated from a subset of metadata associated with the image and (ii) metadata associated with the screen shot of the image. receive a packet containing
extracting, from a packet containing a screenshot of the image displayed by the client device and metadata related to the screenshot of the image, a string and additional metadata related to the screenshot of the image;
decode an identifier from the string using a portion of the additional metadata associated with the screenshot of the image, the identifier including a portion of the metadata associated with the image sheet; And
A watermarking method for tracking delivery of content items associated with images in response to decoded identifiers. 26. The method of claim 25,
The metadata associated with the image includes a timestamp of the image, and the additional metadata includes a timestamp of a screen shot of the image by a client device. 27. The method of claim 25 or 26,
and generating a string from a predetermined number of least significant bits of metadata associated with the image.

Images