WO2023038617A1

WO2023038617A1 - On-device image authentication

Info

Publication number: WO2023038617A1
Application number: PCT/US2021/049255
Authority: WO
Inventors: Yanru Li; Andrea DI BLAS; Olivier Jean BENOIT
Original assignee: Google Llc
Priority date: 2021-09-07
Filing date: 2021-09-07
Publication date: 2023-03-16
Also published as: TW202312105A; EP4168917A1

Abstract

Methods, systems, and apparatus, for on-device image authentication. One of the apparatus includes an image sensor configured to capture raw image data; a processing pipeline having one or more processing elements configured to generate processed image data from the raw image data captured by the image sensor, wherein the processed image data includes an encoded image generated by an encoder; a decoder configured to generate a decoded frame from the processed image data generated from the processing pipeline; and an image authentication module configured to generate an output in an authenticated image format, the output including the processed image data and a representation of a delta between the decoded frame of the processed image data and the raw image data.

Description

ON-DEVICE IMAGE AUTHENTICATION

BACKGROUND

[1] This specification relates to systems and devices for image authentication.

[2] Image authentication is the application of trusted image encoding and decoding to determine if a questioned image is an accurate representation of the original data captured by an imaging sensor, e.g., a camera. Image authentication can be applied to a device or a system that receives and displays an image, such as a camera application in a mobile device, a property monitoring and security device, any social media application that displays images, and so on.

[3] Image authentication plays an important role for risk identification and fraud identification for a variety of use cases. For example, when images of a damaged car are sent to an insurance company during an insurance claim, the insurance company may need to determine whether the images of the damaged car are authentic. As another example, a bank needs to determine whether an image of a check for deposit has been tampered with. A government might want to know whether images passed around in social media are fake images that spread fake news. A user of a social media platform might want to know whether an image is authentic or overly manipulated.

[4] Some image authentication techniques involve generating an authentication image in a specialized image format, which mainly focuses on proving that the raw image data was captured by the camera sensor. The generated authentication image can include (i) raw image data; (ii) a digital signature for a hash of the raw image data; (iii) a processed image of the raw image data in a compressed image format; and (iv) a digital signature for a hash of the processed image of the raw image data. Examples of the specialized image format include Digital NeGative image format (DNG). The provenance of images can also be established using watermarking. For example, an image can include a watermark that identifies the creator or owner of the image.

[5] One drawback of conventional image authentication is that it can be spoofed. For example, if an image has two individual signatures for a raw image and a processed image, one of the images, together with its corresponding signature, can be maliciously modified or replaced. [6] In addition, a conventional authentication image can have a large file size because all the raw data may be included in the authentication image, in addition to the processed version of the image data.

SUMMARY

[7] This specification describes technologies for on-device image authentication. The on-device image authentication techniques described below use an output generated in an authenticated image format that includes a post-processed image and additional information that allows an image rendering computer to reconstruct an earlier version of the image data. The additional information can include a representation of a delta between a decoded frame of the processed image data and the earlier version of image data before processing was applied. The delta represents the difference between the earlier version of the image and the decoded frame and can be used by an image rendering computer to reconstruct the earlier version of the image. In this specification, the earlier version of the image data used to compute the delta will be referred to as raw image data. The raw image data can, but need not, be the first sensor data acquired by a camera before any processing occurs. Rather, raw image data means data from an earlier stage of an image processing pipeline that can be used to compute a delta. In addition, the term image data means any appropriate data that can be used to generate an image rendering, and thus includes actual image formats, as well as other binary information or intermediate representations that can be used for the same purpose.

[8] The subject matter described in this specification can be implemented in particular embodiments so as to realize one or more of the following advantages. Rather than generating two individual signatures for raw image data and processed image data as in a conventional image authentication format, a digital signature in the authenticated image format is generated based on the full content, i.e., both the processed image data and the representation of the delta, to allow for authenticating the generated output in the authenticated image format. That is, the joint signature can ensure that authentication is based on the whole content in the generated output. The authenticated image format results in smaller file size because a representation of a delta between a decoded frame of the processed image data and the raw image data can be saved as a part of the generated output, rather than saving the entire raw image data. Therefore, compared with the conventional image authentication format, the authenticated image format reduces the cost associated with storing and transmitting the generated output in the authenticated image format.

[9] After receiving a request to generate the raw image data from an input data in the authenticated image format, an image rendering computer can reconstruct the raw image data from the representation of the delta between the decoded frame of the processed image data and the raw image data. The image rendering computer can present the processed image data and the raw image through a user interface such that a user can compare the reconstructed raw image data with the processed image data, and the user can determine whether the input data in the authenticated image format is authentic, e.g., whether the processed image data is an authentic representation of the raw image data. In some implementations, the image rendering computer or another computer can receive the processed image data and the raw image as input and can generate, using, for example, a machine learning algorithm, a likelihood score indicating whether the processed image data is an authentic representation of the raw image data.

[10] Moreover, raw image data taken directly from the sensor may not be the best data for image authentication for some of the use cases. In many use cases, postprocessing on the raw image data can be helpful to improve the image quality of the raw image data without compromising the authenticity of the image. For example, image enhancements, e.g., denoising, motion correction, etc., can help improve the image quality. In some cases, multiple raw images can be used to create one processed image data, such as High Dynamic Range (HDR) imaging, low-light imaging, and so on. Therefore, the final post-processed image can be related to multiple raw images, instead of a single raw image. These post-processed images can still be considered as authentic images even though they are different from the raw image data. The authenticated image format can allow authentication of processed images generated from a single raw image or a sequence of multiple raw images.

[11] In addition, the authenticated image format can include a digital certificate that includes a public key of a public-private key pair of the device that generated the input data in the authenticated image format. The image rendering computer can verify whether the device that generated the input data is trustworthy by verifying the public key in the digital certificate, e.g., by verifying the signature in the certificate with a Certification Authority (CA) public key. In some implementations, the image rendering computer can retrieve a device public key stored in a trusted server and can verify the digital signature using the retrieved device public key. This ensures that an untrusted device cannot generate an unauthorized public-private key pair and cannot claim that the unauthorized public-private key pair corresponds to another trusted device.

[12] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

[13] FIG. 1 is a diagram of an example system for generating an output in an authenticated image format.

[14] FIG. 2 is a diagram of an example computing device.

[15] FIG. 3 is a diagram of an example system for rendering an image in an authenticated image format.

[16] FIG. 4 is a flowchart of an example process for generating an output in an authenticated image format.

[17] FIG. 5 is a flowchart of an example process for reconstructing raw image data from input data in an authenticated image format.

[18] Like reference numbers and designations in the various drawings indicate like components.

DETAILED DESCRIPTION

[19] FIG. 1 is a diagram of an example system 100 for generating an output in an authenticated image format.

[20] The system 100 includes a sensor 102, an image processing pipeline 106, a decoder 120, a difference encoder 124, and an authenticated image format generator 128.

[21] The sensor 102 can capture raw image data 104. The sensor 102 can be a camera of the system 100, or any other kind of image capturing sensor. The raw image data 104 can include a raw image, or multiple raw images at a sequence of frames captured within a period of time. The raw image data 104 can include a color image or a grayscale image. The raw image data 104 is not processed by any image post-processing methods. [22] The system 100 can perform a post-processing on the raw image data 104 using an image processing pipeline 106. The image processing pipeline 106 can include a sequence of image processing steps and can generate processed image data 108 with improved image quality and or reduced file size. Each image processing step can use one or more image processing algorithms. In some implementations, the image processing pipeline 106 can include a step that can generate an enhanced image that improves the appearance of the raw image data, such as denoising, motion correction, contrast enhancement, and so on. In some implementations, the image processing pipeline 106 can include a step that generates one enhanced image from multiple raw images at multiple frames, such as HDR/HDR+ imaging, low-light imaging, and so on. In some implementations, the image processing pipeline 106 can include a step that aligns the multiple raw images at multiple frames and merges the multiple raw images into a single image, and the single image can be further enhanced and compressed by other steps of the image processing pipeline 106.

[23] In some implementations, the image processing pipeline 106 can include an encoding step that compresses the raw image data 104 or the enhanced image and generates the processed image data 108 in a compressed image format, e.g., JPEG, PNG, TIFF, and so on. The processed image data 108 can include a processed image, or other format of processed data generated from the raw image data 104. The processed image data 108 in the compressed image format can have a smaller file size than a raw image in the raw image data 104.

[24] Some image processing pipeline 106 can generate processed image data 108 that is an authentic representation of the raw image data 104. For example, an image processing pipeline 106 can generate processed image data 108 using one or more of low-light enhancement, HDR imaging, denoising, deblurring and so on, and the processed image data 108 can still be considered an authentic representation of the raw image data 104.

[25] However, the image processing pipeline 106 can generate processed image data 108 that would not be considered to be an authentic representation of the raw image. The image processing pipeline 106 can, for example, change the shape, size, location, or texture of an object in the image, and/or can add or remove an object in the image. For example, a fake image can be generated by a mobile application that overly modifies the content of the image, e.g., a fake image of a car to be submitted to an insurance company with modifications to the damaged portion of the car. As another example, an image can be edited by a computer software such that the processed image contains one or more objects that do not belong to the raw image, e.g., a fake image of a check for deposit to a bank with a modified bank account number.

[26] The decoder 120 can decode the processed image data 108 and generate a decoded frame 122. The decoder 120 can be deterministic and specified. The same decoder can be used later in an image rendering computer to reconstruct the raw image from the authenticated image. The decoded frame 122 includes image data that is comparable to the format of the raw image data 104. For example, the decoded frame 122 can be in an uncompressed image format. In some implementations, the decoder 120 can be a JPEG image decoder that can generate a decoded image from a JPEG encoded image.

[27] The difference encoder 124 can take the decoded frame 122 of the processed image data and the raw image data 104 as input, and can generate a representation of a delta 126 between the decoded frame 122 of the processed image data 108 and the raw image data 104. The difference encoder 124 can compute the delta, e.g., a difference image, by subtracting the decoded frame 122 from a raw image in the raw image data 104. The difference encoder 124 can generate a representation of the delta 126 based on a compression algorithm that measures differences between image frames. For example, the compression algorithm can be similar to a video compression algorithm, such as MPEG-2, MPEG-4 part 10/H.264, or H.265 algorithms, that measures and compresses the difference between image frames in a video. The video compression algorithm can be adapted to measure and compress the difference between the decoded frame 122 and the raw image data 104.

[28] In some implementations, when the raw image data 104 includes two or more raw image frames, the difference encoder 124 can compare the decoded frame with a selected raw image from the multiple raw image frames. In some implementations, the difference encoder 124 can select a raw image that corresponds to a time point that is the same or closest to the time point of the decoded frame 122.

[29] The authenticated image format generator 128 can combine the processed image data 108 and the representation of the delta 126 of the decoded frame and the raw image data into a single output image file 130 in an authenticated image format. Unlike a conventional image authentication method that saves the raw image data 104 into the output image file, the system 100 can include a much smaller, compressed representation of the delta 126 between the decoded frame 122 of the processed image data 108 and the raw image data 104 in the output image file 130. Therefore, the output image file 130 in the authenticated image format can have a smaller size than image data generated with a conventional image authentication method.

[30] Upon request, an image rendering device can reconstruct the raw image data from the output image file 130 that is received by the image rendering device. The reconstructed raw image data can be compared with the processed image data 108 included in the output image file 130 in order to determine whether the processed image data 108 is an authentic representation of the reconstructed raw image data. More details regarding reconstructing and rendering an image in an authenticated image format are described below in connection with FIG. 3.

[31] In some implementations, the output image file can include a digital signature based on both the processed image data 108 and the representation of the delta 122 between the decoded frame 122 and the raw image data 104. The joint signature can be based on the full content in the authenticated image format, i.e., based on both the processed image data 108 and the representation of the delta 122, and can ensure authenticity of the output image file 130. For example, a fraudulent modification can be applied to a digital signature of a portion of a conventional authenticated image format without being identified because the digital signature of a conventional authenticated image format is only based on either raw image data or processed image data. In contrast, a fraudulent modification cannot be applied to a digital signature of a portion of the authenticated image format because the digital signature of the authenticated image format is generated based on the full content in the authenticated image format. That is, the joint signature can ensure the whole content in the generated output 130 is authentic.

[32] FIG. 2 is a diagram of an example computing device 200 that can generate a camera image in the authenticated image format. The computing device 200 includes a system on a chip (SoC) 204 that can generate an authentication image file, e.g., the output image file 130 in the authenticated image format. That is, generating the authentication image file can be performed on the device through a SoC 204. The FIG. 2 and its description are using SoC as an example device. However, the computing device 200 can be any generic processing device.

[33] The SoC 204 includes a camera control 208 that interacts with a camera sensor 202 of the computing device 200. The camera control 208 is connected to the central processing unit (CPU) 222 of the SoC through a SoC bus 234. The camera control 208 can communicate with the camera driver 216 on the CPU 222. In some implementations, the SoC 204 can include a camera firmware 206 that can communicate with the camera driver 216 through the camera control 208.

[34] The SoC 204 includes a ML accelerator 212. The ML accelerator 212 is connected to the CPU 222 through the SoC bus 234. A ML driver 218 can perform post-processing on an image captured by the camera sensor 202. The ML driver 218 can communicate with the ML accelerator 212 during the process of image postprocessing in order to accelerate the one or more steps in the post-processing pipeline 106. In some implementations, the SoC 204 can include a ML accelerator firmware 210 that can communicate with the ML driver 218 through the ML accelerator 212.

[35] The SoC 204 includes a cryptographic engine 214, which can be a software or hardware module that can generate a digital signature for a hash of data of interest. A cryptographic driver (crypto driver) 220 can communicate with the cryptographic engine 214 through the SoC bus 234. The crypto driver 220 can control the cryptographic engine 214 to generate a digital signature of the full content of the data in the authenticated image format, i.e., the processed image data 108 and the representation of the delta 126 between the decoded frame 122 of the processed image data 108 and the raw image data 104.

[36] The cryptographic engine 214 can have access to the hardware keys 228 through the SoC bus 234. The hardware keys 228 can include a public-private key pair associated with the computing device 200. The public-private key pair can indicate whether the computing device 200 is trustworthy. The cryptographic engine 214 can generate the digital signature of the processed image data 108 and the representation of the delta 126 using a private key of the hardware keys 228 of the computing device 200. For example, the cryptographic engine 214 can generate the digital signature by encrypting a hash value of the processed image data 108 and the representation of the delta 126 using a private key of the hardware keys 228.

[37] The SoC 204 includes a memory control 224 that controls a memory device 230 in the computing device 200. The memory control 224 is connected with the CPU 222 through the SoC bus 234. The memory device 230 can include both volatile and non-volatile memory, such as Random Access Memory (RAM) and Flash RAM. For example, after the camera sensor 202 captures raw image data 104, the raw image data 104 can be saved to a shared buffer in the memory device 230. The CPU can later access the raw image data 104 saved in the memory device 230 and perform post-processing on the raw image data 104 using the ML accelerator 212. The processed image data 108, and the representation of the delta 126 can also be saved in a shared buffer of the memory device 230 that is controlled by the memory control 224.

[38] The SoC 204 includes a storage control 226 that controls storage device 232 in the computing device 200. The storage device 232 can include, for example, one or more of a hard disk device, an optical disk device, a solid state memory device, and so on. The storage control 226 is connected with the CPU 222 through the SoC bus 234. After the SoC 204 generates the output image file 130 in the authenticated image format, the output image file 130 can be saved to the storage device 232 that is controlled by the storage control 226.

[39] Below is an example process of generating an authenticated image file on the computing device 200.

[40] A camera application of the computing device 200 can trigger the camera sensor 202 to capture an image. The camera firmware 206 can communicate with the camera driver 216 on the CPU 222 to save the raw image data 104 to a shared buffer in the memory device 230.

[41] The camera application can invoke an image post-processing component in the SoC 204 to perform post-processing on the raw image data to generate processed image data, e.g., generating a processed image using HDR+ post-processing. For example, the camera application can invoke the ML driver 218 on the CPU 222 to perform post-processing on the raw image data 104 to generate processed image data 108. The CPU 222 can compare the processed image data 108 with the raw image data 104 to generate the difference, i.e., the delta between the decoded frame 122 of the processed image data 108 and the raw image data 104. In some implementations, the ML driver 218 can work with a ML accelerator firmware 210 to perform the image post-processing pipeline and to generate the delta between the two images.

[42] The CPU 222 can perform a difference encoding using a difference encoder 209 to generate a compressed delta of the difference between the raw image data and the decoded frame of the post-processed image data. For example, the difference encoding can be entropy coding, Huffman coding, run-length encoding, and/or any other suitable technique. After generating the compressed delta, the CPU 222 can append the compressed delta to the processed image data. [43] The camera application can invoke the cryptographic driver 220 to control the cryptographic engine 214 to generate a digital signature of the processed image data 108 and a representation of the delta 126 between the decoded frame 122 of the processed image data 108 and the raw image data 104. In some implementations, the cryptographic engine 214 can have access to the hardware keys 228 of the computing device and can generate the digital signature using the private key of the hardware keys 228.

[44] The camera application can generate an output image file 130 in an authenticated image format. The output image file 130 can include the processed image data 108 and the representation of the delta 126. The output image file 130 can further include the digital signature that is generated based on the processed image data 108 and the representation of the delta 126. The camera application can save the output image file 130 into the storage device 232.

[45] FIG. 3 is a diagram of an example system 300 for rendering an image in an authenticated image format. The process of rendering the image in the authenticated image format can be implemented in an operation system of an image rendering computer, can be implemented as a part of a standalone image viewing or editing application of a computing device, or can be implemented as a part of a web browser software, and so on.

[46] An image rendering computer 304 can receive input data 302 in an authenticated image format. The image rendering computer 304 can be a computer that is different from the computing device that generated the input data 302. As described in FIG. 1, the input data 302 in the authenticated image format can be the output image file 130 generated by an on-device image authentication process. The input data 302 can include processed image data 108 and a representation of a delta 126 between a decoded frame of the processed image data and raw image data.

[47] The image rendering computer 304 can receive a request 306 to view the raw image included in the input data 302, e.g., from a user 316 of the image rendering computer 304.

[48] For example, a vehicle owner can capture an image of a damaged vehicle using a mobile device. Because the image is captured at night, the mobile device might perform low-light correction and JPEG encoding, and can generate a processed encoded image of the raw image of the damaged vehicle. The mobile device can generate an image file in the authenticated image format that includes the low-light corrected processed and encoded image of the damaged vehicle and a representation of the delta between the processed image data and the raw image captured by the camera. The vehicle owner can upload the image file in the authenticated image format to a computer of an insurance company during a process of claiming coverage for the damage to the vehicle. The computer of the insurance company, i.e., the image rendering computer 304, can receive the uploaded image file in the authenticated image format. An insurance agent 316 can send a request to the computer to view the raw image included in the uploaded image file in the authenticated image format. For example, the insurance agent 316 may want to visually compare the raw image and the processed image to determine whether the post processed image is an authentic representation of the damaged vehicle. In some implementations, the insurance agent 316 can send a request to the computer and instruct the computer to verify the digital signature included in the uploaded image file.

[49] Upon receiving the request to view the raw image 306, the image rendering computer 304 can generate a processed image 308 and a raw image 310 from the input data 302 in the authenticated image format. The image rendering computer 304 can generate the processed image 308 by reading the processed image included in the input data 302. The image rendering computer 304 can generate a decoded image of the processed image 308 by decoding the processed image using a decoder, e.g., the same as the decoder 120 during the generation of the input data 302. The image rendering computer 304 can generate the raw image 310 by combining the decoded image with the representation of the delta 126 that is included in the input data 302.

[50] For example, a computer of the insurance company can generate a processed image of the damaged vehicle, e.g., a JPEG encoded image of the damaged vehicle, using a JPEG decoder that decodes a JPEG image. The computer can generate a raw image corresponding to the processed image by combining the representation of the delta 126 included in the input data 302 with the decoded image of the processing image 308.

[51] The image rendering computer 304 can include a display device 314, such as a monitor of a desktop computer, a screen of the mobile device, a VR and/or AR equipment and so on. In some implementations, as shown in FIG. 3, the image rendering computer 304 can present the processed image 308 and the raw image 310 side by side on the display device 314. In some implementation, not currently shown in FIG. 3, the image rendering computer 304 can present a user interface element that enables a user 316 to toggle between the processed image 308 and the raw image 310. In some implementations, not currently shown in FIG. 3, the image rendering computer 304 can present a user interface element that displays a difference, e.g., a difference image, between the processed image 308 and the raw image 310. For example, the difference between two images can be calculated by finding the difference between each pixel in each image, and generating an image based on the difference between each pixel in each image.

[52] In some implementations, the image rendering computer 304 can receive a request 306 to verify the digital signature included in the input data 302. In some implementations, the image rendering computer 304 can verify the digital signature before generating the raw image from the input data 302. If the verification of the digital signature is successful, the image rendering computer 304 can proceed to reconstruct the raw image from the input data 302 in the authenticated image format, and the image rendering computer 304 can notify the user 316 that the computing device that generated the input data 302 is a trustworthy device. If the verification of the digital signature is not successful, the image rendering computer 304 may not proceed to reconstruct the raw image, and the image rendering computer 304 can notify the user 316 that the computing device that generated the input data 302 is not a trustworthy device.

[53] In some implementations, the image rendering computer 304 can display a device identity 312 on the display device 314. The device identity 312 can be included in a digital certificate along with the device public key. The digital certificate can be signed by a certification authority (CA) private key, e.g., through a sub-CA, at the device manufacturer. The device identity 312 can include an identification of the device that generated the input data 302, or the verification result of the digital signature, e.g., “Device Verified”, or “Device Not Verified”. A user 316 can review the digital signature verification result included in the device identity 312 and can determine whether the device that generated the input data 302 is trustworthy.

[54] In some implementations, the image rendering computer 104 can include a generic image reader, e.g., a JPEG decoder, that is not specialized to process the input data 302 in the authenticated image format. The image rendering computer 104 can still decode and render the processed image, e.g., rendering the processed JPEG image as if it was just a normal JPEG image. In some implementations, the image rendering computer 104 can discard the rest of the data in the input data 302, e.g., discarding the representation of the delta between the decoded frame of the processed image and the raw image.

[55] In some implementations, the image rendering computer 104 can receive input data that is not in the authenticated image format, e.g., a post-processed image in JPEG format. The image rendering computer 104 can still generate the post-processed image, e.g., by performing a JPEG decoding process, and displays the post-processed image on the displaying device 314, without generating a raw image 310. Therefore, the image rendering computer 304 is backward compatible to display an input data that is not in the authenticated image format.

[56] FIG. 4 is a flowchart of an example process for generating an output in an authenticated image format. For convenience, the process will be described as being performed by a system for generating an output image in an authenticated image format, for example, the system 100 in FIG. 1. The system can include the components of a computing device 200 described in reference to FIG. 2, including one or more camera sensors, one or more SoCs, one or more memory devices, and one or more storage devices, or some combination of these.

[57] The system captures raw image data by an image sensor (402). In some implementations, the raw image data can include a single image of a single raw frame captured by a camera sensor. In some implementations, the raw image data can include a plurality of images of a sequence of raw frames captured by a camera sensor. The system can perform an alignment and merging processing to generate a single image as a raw image that represents what is being captured by the camera.

[58] The system generates processed image data from the new image data captured by the image sensor by a processing pipeline having one or more processing elements (404). The processing pipeline can include one or more image processing routines or algorithms that can generate processed image data with improved image quality. The processed image data can include a processed image, or other format of processed data generated from the raw image data. In some implementations, the processed image can be in a compressed image format, e.g., JPEG, PNG, TIFF, or another compressed image format.

[59] The system generates a decoded frame from the processed image data generated from the processed pipeline (406). In some implementations, the decoded frame can be in an uncompressed image format. For example, when the processed image is in JPEG format, the system can generate a decoded frame from the processed image using a JPEG image decoder.

[60] The system generates an output in an authenticated image format (408). The output includes the processed image data and a representation of a delta between the decoded frame of the processed image data and the raw image data. That is, instead of saving the raw image data, the system generates the difference between the raw image data and the decoded frame of the processed image data. Then, the system encodes the difference in the output image in the authenticated image format, such that an image reader can later properly reconstruct the raw image data upon request.

[61] The system can reliably generate and encode the raw image frame independently of the number and type of processing steps in the processing pipeline that is used to generate the processed image data. Later, an image reader that is configured to process the output file in the authenticated image format can reconstruct and display the raw image frame.

[62] In some implementations, the representation of the delta between the decoded frame of the processed image data and the raw image data can be generated based on a compression algorithm that measures differences between image frames. The compression algorithm is efficient in processing the differences between image frames, e.g., the differences that have lots of zeros. When two image frames are sufficiently similar with each other, the difference between the two image frames can be compressed efficiently. For example, the system can generate the output using a video encoding algorithm that can encode the difference between frames in a video, such as a video MPEG encoder.

[63] In some implementations, the output can include a digital signature based on both the processed image data and the representation of the delta between the decoded frame of the processed image data and the raw image data. That is, the system can sign and authenticate both the processed and encoded frame as well as the difference information, i.e., the delta between the decoded frame of the processed image data and the raw image data, in the same output file.

[64] In some implementations, the system can generate a hash value based on the processed image data and the representation of the delta between the decoded frame of the processed image data and the raw image data, and the system can generate the digital signature based on the hash value. A hash value is a numerical value of a fixed length that uniquely identifies data. A hash value can represent a large amount of data as a much smaller numeric value.

[65] In some implementations, the system can generate the digital signature by encrypting the hash value using a private key of a public-private key pair of the device. The system can have access to the hardware keys associated with the computing device of the system, e.g., the hardware keys 228 of the commuting device 200 in FIG. 2. The public-private key pair can indicate whether the computing device is trustworthy. Later, an image rendering computer can obtain a public key of the pair of private keys associated with the device and can verify the digital signature in the output file in the authenticated image format in order to determine whether the device that generated the output file is trustworthy.

[66] In some implementations, the output can include a digital certificate that includes a public key of a public-private key pair of the device. In some implementations, the output can further include a device identifier, validity date, and a digital signature generated by a CA or a sub-CA. Later, an image rendering computer can verify the digital certificate in the output file in the authenticated image format using the public key in order to determine whether the device that generated the output file is trustworthy.

[67] FIG. 5 is a flowchart of an example process for reconstructing raw image data from input data in an authenticated image format. For convenience, the process will be described as being performed by a system for image rendering, for example, the system 300 in FIG. 3. The system can include an image rendering computer or device that can be configured to parse the input data in the authenticated image format and reconstruct the raw image data from the input data in the authenticated image format.

[68] The system receives input data in an authenticated image format (502). The input data includes processed image data and a representation of delta between a decoded frame of the processed image data and raw image data. The processed image data can include a processed image, or other format of processed data generated from the raw image data. For example, the input data can include a post-processed image in JPEG format and a delta between a decoded frame of the post-processed image and a raw image captured by a camera sensor.

[69] The system receives a request to generate the raw image data from the input data (504). For example, the system can receive a request 306 from a user of the image rendering system to display the raw image 310 and the processed image 308, such that the user can verify the authenticity of the input data 302.

[70] The system reconstructs the raw image data from the representation of the delta between the decoded frame of the processed image data and the raw image data (506). In some implementations, reconstructing the raw image data from the representation of the delta between the decoded frame of the processed image data and the raw image data can include generating the raw image data by combining the processed image data with the representation of the delta between the decoded frame of the processed image data and the raw image data.

[71] In some implementations, the input data in the authenticated image format can include a digital signature generated based on the full content, i.e., both the processed image data and representation of the delta between the decoded frame of the processed image data and the raw image data. The digital signature generated on the full content can ensure the authenticity of the input data. That is, the digital signature can ensure that the processed image data and the representation of the delta cannot be modified after the digital signature is computed.

[72] In some implementations, the digital signature included in the input data can be generated by (i) generating a hash value based on the processed image data and the representation of the delta between the decoded frame of the processed image data and the raw image data, and (ii) encrypting the hash value using a private key of a public-private key pair associated with a device that generates the input data in the authenticated image format. In some implementations, the public-private key pair can be hardware keys associated with the device that generated the input data, e.g., the hardware keys 228 of the computing device 200. The digital signature can indicate whether the input data was generated by a trustworthy device that is authorized to have access to the private key. In some implementations, the system can obtain a public key of the public-private key pair associated with the device that generates the input in the authenticated image format, and the system can verify the digital signature in the input data using the public key.

[73] In some implementations, the input data in the authenticated image format can include a digital certificate that can be used to verify whether the input data was generated by a trustworthy device. The system can, before verifying the digital signature in the input data in the authenticated image format, verify the digital certificate in the input data, e.g., verifying that the input data was generated by a trustworthy device. Although any device can provide an authenticated image with a valid signature, only a legitimate device can have its device public key certified, e.g., by including the device public key in a valid digital certificate. An invalid device manufacturer cannot get its public key certified, e.g., by a CA or by a secure server.

[74] In response to a successful verification of the digital certificate, the system can extract the device public key from the digital certificate. Then the system can proceed to verify the validity of the input data in the authenticated image format by verifying the digital signature using the device public key. If the input data is valid, the system can reconstruct the raw image data from the input data in the authenticated image format and display the reconstructed raw image to a user such that the user can determine whether the processed image is an accurate representation of the reconstructed raw image. In some implementations, the system can provide the reconstructed raw image and the processed image to a computer-implemented algorithm, e.g., a machine learning algorithm, and the algorithm can generate a matching score between the reconstructed raw image and the processed image, and the matching score can represent a likelihood that the processed image is an accurate representation of the reconstructed raw image.

[75] In some implementations, the device public key can be provided in a digital certificate. The digital certificate can be certified by a certification authority (CA), and can be included in the input data in the authenticated image format. The digital certificate can be generated outside the device and can be provisioned in the device at the time of manufacturing. The device public key associated with a private key of the device can be certified by the CA. The system can verify the digital certificate included in the input data with a CA root public key in order to ensure the trustworthiness of the device that generated the input data in the authenticated image format. For example, the system can obtain the digital certificate by accessing metadata included in the input data, and the system can verify the digital certificate with a CA root public key.

[76] In some implementations, the device public key can be stored in a trusted server or a trusted database that can be accessed by the system, and the system can obtain the device public key stored in the trusted server and can verify the digital signature in the input data using the device public key. For example, the system can retrieve the device public key stored in a trusted server based on an identification (ID) that is provided in the metadata of the input data. The system can verify the digital signature with the device public key retrieved from the trusted server.

[77] In some implementations, the system can present the processed image data and the reconstructed raw image data side by side on a displaying device of the system. In some implementations, the system can present a user interface element that toggles between the processed image data and the reconstructed raw image data. For example, a user of the system can use the user interface element to toggle between the processed image and the reconstructed raw image.

[78] In some implementations, the system or another system can take as input the processed image data and the reconstructed raw image data and generate a prediction result indicating whether the processed image data is an authentic representation of the reconstructed raw image data. In some implementations, the system or the other system can use a trained machine learning model to generate a prediction result indicating the likelihood that the processed image data is an authentic representation of the reconstructed raw image data.

[79] Embodiments of the subject matter and the actions and operations described in this specification can be implemented in digital electronic circuitry, in tangibly- embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non-transitory storage medium for execution by, or to control the operation of, data processing apparatus. Alternatively or in addition, the program instructions can be encoded on an artificially-generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus. The computer storage medium can be or be part of a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. A computer storage medium is not a propagated signal.

[80] A computer program, which may also be referred to or described as a program, software, a software application, an app, a module, a software module, an engine, a script, or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and it can be deployed in any form, including as a stand-alone program or as a module, component, engine, subroutine, or other unit suitable for executing in a computing environment, which environment may include one or more computers interconnected by a data communication network in one or more locations.

[81] A computer program may, but need not, correspond to a file in a file system. A computer program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub-programs, or portions of code.

[82] To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on, or configured to communicate with, a computer having a display device, e.g., a LCD (liquid crystal display) monitor, for displaying information to the user, and an input device by which the user can provide input to the computer, e.g., a keyboard and a pointing device, e.g., a mouse, a trackball or touchpad. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user’s device in response to requests received from the web browser, or by interacting with an app running on a user device, e.g., a smartphone or electronic tablet. Also, a computer can interact with a user by sending text messages or other forms of message to a personal device, e.g., a smartphone that is running a messaging application, and receiving responsive messages from the user in return.

[83] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back-end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front-end component, e.g., a client device having a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN) and a wide area network (WAN), e.g., the Internet.

[84] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server transmits data, e.g., an HTML page, to a user device, e.g., for purposes of displaying data to and receiving user input from a user interacting with the device, which acts as a client. Data generated at the user device, e.g., a result of the user interaction, can be received at the server from the device.

[85] In addition to the embodiments described above, the following embodiments are also innovative:

[86] Embodiment 1 is a device comprising: an image sensor configured to capture raw image data; a processing pipeline having one or more processing elements configured to generate processed image data from the raw image data captured by the image sensor, wherein the processed image data comprises an encoded image generated by an encoder; a decoder configured to generate a decoded frame from the processed image data generated from the processing pipeline; and an image authentication module configured to generate an output in an authenticated image format, the output comprising the processed image data and a representation of a delta between the decoded frame of the processed image data and the raw image data.

[87] Embodiment 2 is the device of embodiment 1, wherein the raw image data can be reconstructed based on the delta and the processed image data.

[88] Embodiment 3 is the device of embodiment 1, wherein the output comprises a digital signature based on both the processed image data and the representation of the delta between the decoded frame of the processed image data and the raw image data.

[89] Embodiment 4 is the device of embodiment 3, wherein generating the digital signature comprises: generating a hash value based on the processed image data and the representation of the delta between the decoded frame of the processed image data and the raw image data; and generating the digital signature based on the hash value.

[90] Embodiment 5 is the device of any one of embodiments 3-4, wherein generating the digital signature comprises encrypting the hash value using a private key of a public-private key pair of the device.

[91] Embodiment 6 is the device of embodiment 3, wherein the output comprises a digital certificate, wherein the digital certificate comprises a public key of a publicprivate key pair of the device.

[92] Embodiment 7 is the device of embodiment 1, wherein the encoded image is in a compressed image format.

[93] Embodiment 8 is the device of embodiment 7, wherein the decoded frame is in an uncompressed image format.

[94] Embodiment 9 is the device of embodiment 1, wherein the representation of the delta between the decoded frame of the processed image data and the raw image data is generated based on a compression algorithm that measures differences between image frames.

[95] Embodiment 10 is a computer implemented method, comprising: capturing raw image data by an image sensor; generating processed image data from the raw image data captured by the image sensor by a processing pipeline having one or more processing elements, wherein the processed image data comprises an encoded image generated by an encoder; generating a decoded frame from the processed image data generated from the processing pipeline; and generating an output in an authenticated image format, the output comprising the processed image data and a representation of a delta between the decoded frame of the processed image data and the raw image data.

[96] Embodiment 11 is one or more non-transitory storage media encoded with instructions that when executed by a computing device cause the computing device to perform operations comprising: capturing raw image data by an image sensor; generating processed image data from the raw image data captured by the image sensor by a processing pipeline having one or more processing elements, wherein the processed image data comprises an encoded image generated by an encoder; generating a decoded frame from the processed image data generated from the processing pipeline; and generating an output in an authenticated image format, the output comprising the processed image data and a representation of a delta between the decoded frame of the processed image and the raw image data.

[97] Embodiment 12 is a computer implemented method, comprising: receiving, by an image rendering computer, an input data in an authenticated image format, the input data comprising processed image data and a representation of a delta between a decoded frame of the processed image data and raw image data; receiving a request to generate the raw image data from the input data; and reconstructing the raw image data from the representation of the delta between the decoded frame of the processed image data and the raw image data.

[98] Embodiment 13 is the method of embodiment 12, wherein reconstructing the raw image data from the representation of the delta between the decoded frame of the processed image data and the raw image data comprises generating the raw image data by combining the processed image data with the representation of the delta between the decoded frame of the processed image data and the raw image data.

[99] Embodiment 14 is the method of embodiment 12, wherein the input data in the authenticated image format comprises a digital signature generated based on both the processed image data and the representation of the delta between the decoded frame of the processed image data and the raw image data, and wherein the method comprises: before reconstructing the raw image data from the input data in the authenticated image format, verifying the digital signature in the input data; and in response to a successful verification of the digital signature, reconstructing the raw image data from the input data in the authenticated image format.

[100] Embodiment 15 is the method of embodiment 14, wherein the digital signature is generated by encrypting a hash value using a private key of a public-private key pair associated with a device that generates the input data in the authenticated image format, and wherein verifying the digital signature in the input data comprises: obtaining a public key of the public-private key pair associated with the device that generates the input data in the authenticated image format; and verifying the digital signature in the input data using the public key.

[101] Embodiment 16 is the method of embodiment 15, wherein the public key is certified by a certification authority (CA), the input data in the authenticated image format comprises a digital certificate, wherein obtaining the public key and verifying the digital signature in the input data using the public key comprises: verifying, by the image rendering computer, the digital certificate comprised in the input data with a CA root public key; in response to a successful verification, obtaining, by the image rendering computer, the public key comprised in the digital certificate; and verifying, by the image rendering computer, the digital signature in the input data using the public key.

[102] Embodiment 17 is the method of embodiment 15, wherein the public key is stored in a trusted server that can be accessed by the image rendering computer, wherein obtaining the public key and verifying the digital signature in the input data using the public key comprises: obtaining, by the image rendering computer, the public key stored in the trusted server; and verifying, by the image rendering computer, the digital signature in the input data using the public key.

[103] Embodiment 18 is the method of embodiment 12, further comprising: presenting the processed image data and the raw image data side by side.

[104] Embodiment 19 is the method of embodiment 12, further comprising: presenting a user interface element that toggles between the processed image data and the raw image data.

[105] Embodiment 20 is the method of embodiment 12, further comprising: presenting a user interface element that displays a difference between the processed image data and the raw image. [106] Embodiment 21 is one or more non-transitory storage media encoded with instructions that when executed by a computing device cause the computing device to perform operations comprising: receiving an input data in an authenticated image format, the input data comprising processed image data and a representation of a delta between a decoded frame of the processed image data and raw image data; receiving a request to generate the raw image data from the input data; and reconstructing the raw image data from the representation of the delta between the decoded frame of the processed image data and the raw image data.

[107] Embodiment 22 is a system comprising one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising: receiving an input data in an authenticated image format, the input data comprising processed image data and a representation of a delta between a decoded frame of the processed image data and raw image data; receiving a request to generate the raw image data from the input data; and reconstructing the raw image data from the representation of the delta between the decoded frame of the processed image data and the raw image data.

[108] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or on the scope of what is being or may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially be claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claim may be directed to a subcombination or variation of a subcombination.

[109] Similarly, while operations are depicted in the drawings and recited in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[HO] Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous.

[Hl] What is claimed is:

Claims

1. A device comprising: an image sensor configured to capture raw image data; a processing pipeline having one or more processing elements configured to generate processed image data from the raw image data captured by the image sensor, wherein the processed image data comprises an encoded image generated by an encoder; a decoder configured to generate a decoded frame from the processed image data generated from the processing pipeline; and an image authentication module configured to generate an output in an authenticated image format, the output comprising the processed image data and a representation of a delta between the decoded frame of the processed image data and the raw image data.

2. The device of claim 1, wherein the raw image data can be reconstructed based on the delta and the processed image data.

3. The device of claim 1, wherein the output comprises a digital signature based on both the processed image data and the representation of the delta between the decoded frame of the processed image data and the raw image data.

4. The device of claim 3, wherein generating the digital signature comprises: generating a hash value based on the processed image data and the representation of the delta between the decoded frame of the processed image data and the raw image data; and generating the digital signature based on the hash value.

5. The device of any one of claims 3-4, wherein generating the digital signature comprises encrypting the hash value using a private key of a public-private key pair of the device.

26

6. The device of claim 3, wherein the output comprises a digital certificate, wherein the digital certificate comprises a public key of a public-private key pair of the device.

7. The device of claim 1, wherein the encoded image is in a compressed image format.

8. The device of claim 7, wherein the decoded frame is in an uncompressed image format.

9. The device of claim 1, wherein the representation of the delta between the decoded frame of the processed image data and the raw image data is generated based on a compression algorithm that measures differences between image frames.

10. A computer implemented method, comprising: capturing raw image data by an image sensor; generating processed image data from the raw image data captured by the image sensor by a processing pipeline having one or more processing elements, wherein the processed image data comprises an encoded image generated by an encoder; generating a decoded frame from the processed image data generated from the processing pipeline; and generating an output in an authenticated image format, the output comprising the processed image data and a representation of a delta between the decoded frame of the processed image data and the raw image data.

11. One or more non-transitory storage media encoded with instructions that when executed by a computing device cause the computing device to perform operations comprising: capturing raw image data by an image sensor; generating processed image data from the raw image data captured by the image sensor by a processing pipeline having one or more processing elements, wherein the processed image data comprises an encoded image generated by an encoder; generating a decoded frame from the processed image data generated from the processing pipeline; and generating an output in an authenticated image format, the output comprising the processed image data and a representation of a delta between the decoded frame of the processed image and the raw image data.

12. A computer implemented method, comprising: receiving, by an image rendering computer, an input data in an authenticated image format, the input data comprising processed image data and a representation of a delta between a decoded frame of the processed image data and raw image data; receiving a request to generate the raw image data from the input data; and reconstructing the raw image data from the representation of the delta between the decoded frame of the processed image data and the raw image data.

13. The method of claim 12, wherein reconstructing the raw image data from the representation of the delta between the decoded frame of the processed image data and the raw image data comprises generating the raw image data by combining the processed image data with the representation of the delta between the decoded frame of the processed image data and the raw image data.

14. The method of claim 12, wherein the input data in the authenticated image format comprises a digital signature generated based on both the processed image data and the representation of the delta between the decoded frame of the processed image data and the raw image data, and wherein the method comprises: before reconstructing the raw image data from the input data in the authenticated image format, verifying the digital signature in the input data; and in response to a successful verification of the digital signature, reconstructing the raw image data from the input data in the authenticated image format.

15. The method of claim 14, wherein the digital signature is generated by encrypting a hash value using a private key of a public-private key pair associated with a device that generates the input data in the authenticated image format, and wherein verifying the digital signature in the input data comprises: obtaining a public key of the public-private key pair associated with the device that generates the input data in the authenticated image format; and verifying the digital signature in the input data using the public key.

16. The method of claim 15, wherein the public key is certified by a certification authority (CA), the input data in the authenticated image format comprises a digital certificate, wherein obtaining the public key and verifying the digital signature in the input data using the public key comprises: verifying, by the image rendering computer, the digital certificate comprised in the input data with a CA root public key; in response to a successful verification, obtaining, by the image rendering computer, the public key comprised in the digital certificate; and verifying, by the image rendering computer, the digital signature in the input data using the public key.

17. The method of claim 15, wherein the public key is stored in a trusted server that can be accessed by the image rendering computer, wherein obtaining the public key and verifying the digital signature in the input data using the public key comprises: obtaining, by the image rendering computer, the public key stored in the trusted server; and verifying, by the image rendering computer, the digital signature in the input data using the public key.

18. The method of claim 12, further comprising: presenting the processed image data and the raw image data side by side.

19. The method of claim 12, further comprising: presenting a user interface element that toggles between the processed image data and the raw image data.

20. The method of claim 12, further comprising: presenting a user interface element that displays a difference between the processed image data and the raw image.

29

21. One or more non-transitory storage media encoded with instructions that when executed by a computing device cause the computing device to perform operations comprising: receiving an input data in an authenticated image format, the input data comprising processed image data and a representation of a delta between a decoded frame of the processed image data and raw image data; receiving a request to generate the raw image data from the input data; and reconstructing the raw image data from the representation of the delta between the decoded frame of the processed image data and the raw image data.

22. A system comprising one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising: receiving an input data in an authenticated image format, the input data comprising processed image data and a representation of a delta between a decoded frame of the processed image data and raw image data; receiving a request to generate the raw image data from the input data; and reconstructing the raw image data from the representation of the delta between the decoded frame of the processed image data and the raw image data.

30