WO2023225999A9

WO2023225999A9 - Method and apparatus for certifying defense against image transformation

Info

Publication number: WO2023225999A9
Application number: PCT/CN2022/095571
Authority: WO
Inventors: Jun Zhu; Zhongkai HAO; Chengyang YING; Yinpeng DONG; Hang SU; Jian Song; Ze CHENG
Original assignee: Robert Bosch Gmbh; Tsinghua University
Priority date: 2022-05-27
Filing date: 2022-05-27
Publication date: 2024-02-01
Also published as: WO2023225999A1

Abstract

A method for certifying defense against image transformation is disclosed. The method comprises generating a surrogate neural network consisting of three individual neural networks to simulate the image transformation, wherein a first neural network is an encoder for transformation parameters, a second neural network is an encoder for input images, a third neural network is a transformation network for outputs of the first and second neural networks, and the transformation parameters are augmented in dimension and added to the surrogate neural network. The method further comprises certifying the defense against the image transformation by using the surrogate neural network based on randomized smoothing.

Description

METHOD AND APPARATUS FOR CERTIFYING DEFENSE AGAINST IMAGE TRANSFORMATION

FIELD

The present disclosure relates generally to artificial intelligence technology, and more particularly, to techniques on certified defense for image transformation.

BACKGROUND

Deep learning models are widely used in computer vision field, enabling computers and systems to derive meaningful information from digital images, videos and other visual inputs and take actions or make recommendations based on that information. However, deep learning models are vulnerable to adversarial examples including semantic transformed examples, resulting in that applications of such deep learning models in various security-sensitive tasks. For example, a small adversarial patch on the road markings can mislead the autonomous driving system, which raises severe safety concerns. Therefore, it is necessary to certify the robustness of the deep learning models against these adversarial attacks.

Certified defensing methods such as randomized smoothing have shown promise towards building reliable machine learning systems against l _p-norm bounded attacks. However, existing methods are insufficient or unable to provably defend against semantic transformations, especially those without closed-form expressions (such as defocus blur and pixelate) , which are more common in practice and often unrestricted. Compared with the l _p-norm bounded adversarial examples, semantic transformtions can occur more naturally in real-world scenarios, including image rotation, translation, blur, weather, etc., most of which are common corruptions. Such transformations do not damage the semantic features of images that can still be recognized by humans, but they degrade the performance of deep learning models significantly. Therefore, it is imperative and challenging to improve model robustness against these semantic transformations.

SUMMARY

The following presents a simplified summary of one or more aspects according to the present disclosure in order to provide a basic understanding of such aspects. This summary is not an extensive overview of all contemplated aspects, and is intended to neither identify key or critical elements of all aspects nor delineate the scope of any or all aspects. Its sole purpose is to present some concepts of one or more aspects in a simplified form as a prelude to the more detailed description that is presented later.

In an aspect of the disclosure, a method for certifying defense against image transformation is disclosed. The method comprises generating a surrogate neural network consisting of three individual neural networks to simulate the image transformation, wherein a first neural network is an encoder for transformation parameters, a second neural network is an encoder for input images, a third neural network is a transformation network for outputs of the first and second neural networks, and the transformation parameters are augmented in dimension and added to the surrogate neural network; and certifying the defense against the image transformation by using the surrogate neural network based on randomized smoothing.

In another aspect of the disclosure, an apparatus for certifying defense against image transformation is disclosed. The apparatus comprises a surrogate neural network consisting of three individual neural networks for simulating the image transformation, wherein a first neural network is an encoder for transformation parameters, a second neural network is an encoder for input images, a third neural network is a transformation network for outputs of the first and second neural networks, and the transformation parameters are augmented in dimension and added to the surrogate neural network. The surrogate neural network is used to certify the defense against the image transformation based on randomized smoothing.

In another aspect of the disclosure, an apparatus for certifying defense against image transformation is disclosed. The apparatus may comprise a memory and at least one processor coupled to the memory. The at least one processor may be configured to generate a surrogate neural network consisting of three individual neural networks to simulate the image transformation, wherein a first neural network is an encoder for transformation parameters, a second neural network is an encoder for input images, a third neural network is a transformation network for outputs of the first and second neural networks, and the transformation parameters are augmented in dimension and added to the surrogate neural network; and certify the defense against the image transformation by using the surrogate neural network based on randomized smoothing.

In another aspect of the disclosure, a computer readable medium storing computer code for certifying defense against image transformation is disclosed. The computer code, when executed by a processor, may cause the processor to generate a surrogate neural network consisting of three individual neural networks to simulate the image transformation, wherein a first neural network is an encoder for transformation parameters, a second neural network is an encoder for input images, a third neural network is a transformation network for outputs of the first and second neural networks, and the transformation parameters are augmented in dimension and added to the surrogate neural network; and certify the defense against the image transformation by using the surrogate neural network based on randomized smoothing.

In another aspect of the disclosure, a computer program product for certifying defense against image transformation is disclosed. The computer program product may comprise processor executable computer code for generating a surrogate neural network consisting of three individual neural networks to simulate the image transformation, wherein a first neural network is an encoder for transformation parameters, a second neural network is an encoder for input images, a third neural network is a transformation network for outputs of the first and second neural networks, and the transformation parameters are augmented in dimension and added to the surrogate neural network; and certifying the defense against the image transformation by using the surrogate neural network based on randomized smoothing.

Other aspects or variations of the disclosure will become apparent by consideration of the following detailed description and accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

The following figures depict various embodiments of the present disclosure for purposes of illustration only. One skilled in the art will readily recognize from the following description that alternative embodiments of the methods and structures disclosed herein may be implemented without departing from the spirit and principles of the disclosure described herein.

FIG. 1 illustrates a certified defense against semantic transformation in accordance with one aspect of the present disclosure.

FIG. 2 is a graphical illustration of a method for certifying defense against image transformation in accordance with one aspect of the present disclosure.

FIG. 3 illustrates a flow chart of a method for certifying defense against image transformation in accordance with one aspect of the present disclosure.

FIG. 4 illustrates a block diagram of an apparatus for certifying defense against image transformation in accordance with one aspect of the present disclosure.

DETAILED DESCRIPTION

Before any embodiments of the present disclosure are explained in detail, it is to be understood that the disclosure is not limited in its application to the details of construction and the arrangement of features set forth in the following description. The disclosure is capable of other embodiments and of being practiced or of being carried out in various ways.

Although various methods can empirically improve model robustness against semantic transformations on typical benchmarks (evaluated in average-case) , these methods often fail to defend against adaptive attacks by generating adversarial semantic transformations, which are optimized over the parameter space of transformations for the worst-case. In contrast, the certified defenses aim to provide a certified region where a deep learning model is theoretically robust under any attack or perturbation.

FIG. 1 is an illustration of certified defense against semantic transformations in accordance with one aspect of the present disclosure. In this example, the semantic transformations are snow, and a parameter α is used to denote the severity of the transformations on an image of a “STOP” sign. As shown by the dashed circle in the diagram 100, which shows decision boundaries of a model (i.e. a classifier) trained for identifying the “STOP” sign under semantic transformations of snow, the severity of the transformation increases as the radius of the dashed circle becomes greater. Image 110 is a transformed image with ||α||=0.3. Image 120 is a transformed image with ||α||=0.5.

Images

130 and 140 are two different transformed images with ||α||=1.0. Region 150 is an empirical robust region, and region 160 is a certified robust region with a radius of ||α||=0.5. Accordingly, as shown in FIG. 1, the model is certifiably robust with ||α||=0.3 (such as, the transformed image 110) and ||α||=0.5 (such as, the transformed image 120) . Nevertheless, the model may make an erroneous prediction when ||α||=1.0 (such as, against the transformed image 140) , although it may be empirically robust to other corrupted images with ||α||=1.0 (such as, the transformed image 130) . This implies that the empirical robustness may not be sufficient for safety-sensitive applications such as autonomous driving. Autonomous driving cars may encounter any levels and types of complex semantic transformations or corruptions in practice. An ideal safe model should tell users its safe regions that the model is certifiably robust under different transformations.

Unlike an l _p perturbation adding a small amount of noise to an image, semantic attacks are usually unrestricted. For example, the semantic attacks may include the adversarial patches, and the manipulation based on spatial transformations, such as rotation or translation. A wide variety of image corruptions and perturbations degrade the performance of many deep learning models. Most of them such as types of blur, pixelate are hard to be analyzed, such that defending against them is highly challenging. In especial, many non-resolvable transformations, such as zoom blur and pixelate, do not have closedform expressions. This makes the theoretical analyses of these transformations difficult and sometimes impossible with the existing methods, although they are common in real-world scenarios.

Several recent studies have attempted to extend the certified defenses to simple semantic transformations with good mathematical properties like translation, Gaussian blur, and geometric transformations. Randomized smoothing is a recent certification method, which can be used to certify attacks beyond l _p-norm, and can be extended to certify some simiple semantic transformations, such as, image translation and rotation. However, these methods are limited to simple semantic transformations, which are easy to analyze due to their resolvable mathematical properties. These methods are neither scalable nor capable of certifying robustness against complex image corruptions and transformations, especially the non-resolvable ones. Therefore, it remains highly challenging to certify robustness against these complex and realistic semantic transformations. There exists a need for scalable algorithms for certifying most non-resolvable and complex semantic transformations.

In this disclosure, a generalized randomized smoothing (GRS) method for certified robustness against general image transformations, including both the resolvable semantic transformations (e.g., translation) and the non-resolvable semtantic transformations (e.g., rotational blur) . In the GRS method, an surrogate image-to-image translation neural network is generated and used to approximate these image transformations. Due to the strong capacity of neural networks, this method is flexible and scalable for modeling the complex non-resolvable semantic transformations. Then, a certified radius for the surrogate neural network may be calculated by introducing new augmented noise in the layers of the surrogate neural network, which can be used for certifying the original image transformations. It can be proved that the impact of the approximation error on the certified bound can be ignored in practice. After applying the GRS method on several publicly available datasets, the results demonstrate that the method is effective for certifying complex semantic transformations, and may achieve state-of-the-art performance in both certified accuracy and empirical accuracy for different types of image transformations.

For the purpose of easy description, following mathematical notations and formulations are introduced. It can be understood that each of these notations and formulations may have specific physical means in different application scenes. For example, an input

may refer to images captured by autonomous driving cars, n is the dimension of the images, which may dependent on the parameters such as resolution and chromaticity of the images. In this example, the label Y = {1, 2, ..., K} may include cars, buildings, pedestrians, various of signposts (such as, a “STOP” sign as shown in FIG. 1) , and so on. A base model, such as a classifier, may be denoted as f (x) :

which may output predicted probabilities over all K classes or labels for the input images x. The prediction of f is argmax _i∈Y f (x) _i, where f (·) _i may denote the i-th element of f (·) . The image transformation (such as, semantic transformation) of the raw input image x with transformation parameter

may be denoted as τ (θ, x) :

wherein m is the dimension of the parameter θ.

Since the generalized randomized smoothing method in accordance with one aspect of the present disclosure is developed from and based on the randomized smoothing method, the randomized smoothing method will be described first below.

Given an above base classifer f (x) and image transformations τ (θ, x) , a smoothed classifier may be denoted as:

which is the average prediction for the input samples under a smooth distribution g (θ) ∝ exp (-ψ (θ) ) , here ψ :

is a smooth function. y _A is a predicted label of the smoothed classifier G (x) for a clean image, and G (x) _Adenotes the probability of the top-1 class y _A herein. Similarly, y _B is defined as the runner-up (i.e., the second top) class of the smoothed classifier G (x) , and G (x) _B denotes the probability of the class y _B as follows.

Then, a classifier may have a certified robust radius R, if it satisfies that for any perturbation ||ξ|| ≤ R where ||·|| is any l _p norm without specification,

In this disclosure, semantic transformations are categorized into two classes: resolvable transformations and non-resolvable transformations. A semantic transformation is resolvable, if the composition of two transformations with parameters belonging to a perturbation set θ,

is still a transformation with a new parameter

here γ (·, ·) : P × P → P is a function depending on these parameters, i.e., satisfying

τ (θ, τ (ξ, x) ) =τ (γ (θ, ξ) , x) . (4)

Otherwise, the semantic transformation is non-resolvable. Examples of resolvable semantic transformation may include Gaussian blur, translation, brightness, contrast, etc. Examples of non-resolvable semantic transformation may include rotation, scaling, rotational blur, defocus blur, zoom blur, pixelate, etc. The properties of resolvable transformations make it much easier to derive the certified bound.

In order to analysis certified bound for resolvable transformations, a function Φ(·) which will be used in the certified bound may be introduced. For any vector u with unit norm, i.e., ||u|| = 1,

is set as a random variable, where δ ～ g (·) and

is the gradient operator. A complementary Cumulative Distribution Function (CDF) of γ _u is further defined as

and the inverse complementary CDF of γ _u is defined as

The function Φ may be defined as

Then, for resolvable semantic transformations, it (referred as theorem 1 herein) can be proved that: for any classifer f (x) with a corresponding randomized smoothed classifer G(x) defined in equation (1) , if there exists a function M (·, ·) :

satisfying

and there exist two constantsp _A,

satisfying

then

holds for any ||ξ|| ≤ R, where

and M ^*=max _ξ, θ||M (ξ, θ) ||.

For examples, additive transformations and commutable transformations are two types of simple resolvable semantic transformations. A transformation is additive if τ (θ, τ (ξ, x) ) = τ (ξ+θ, x) for any θ, ξ ∈ P. A transformation is commutable if τ (θ, τ (ξ, x) ) = τ (ξ, τ (θ, x) ) for any θ, ξ ∈ P. For these two types of transformations, it may be straightforward to verify that they satisfy the property proposed in Theorem 1 with M (θ, ξ) = I. Consequently, theorem 1 may be simply applied for an isotropic Gaussian distribution g (θ) = N (0, σ ²I) , obtaining a certified radius as

where Ψ is the inverse CDF of the standard Gaussian distribution. These two types of transformations may include image translation and Gaussian blur, which are basic semantic transformations. The certification of these simple transformations only requires applying translation or Gaussian blur to the image sample and we obtain the average classification score under the noise distribution.

However, in practice, most semantic transformations are not commutable or even not resolvable. The existing methods like Semanify-NN based on convex relaxation, and TSS based on randomized smoothing, require developing a specific algorithm or bound for each individual semantic transformation. They are not scalable and might be infeasible for more complicated transformations without explicit mathematical forms. Therefore, it is needed to develop better and more general methods for certifying more types of semantic transformations.

FIG. 2 is a graphical illustration of a generalized randomized smoothing method for certifying defense against image transformations in accordance with one aspect of the present disclosure. As shown in FIG. 2, a surrogate neural network 200 may be generated and used to simulate semantic transformations. The neural network 200 are able to approximate functions including complex and non-resolvable semantic transformations. For example, the neural network 200 may be an image-to-image translation network accurately fitting a semantic transformation.

In one embodiment, the neural network 200 simulating a semantic transformation τ (θ, x) may be defined as the following form which will lead to a simple certified bound:

τ (θ, x) =H (F ₁ (θ) +F ₂ (x)), (8)

where F ₁ (·) , F ₂ (·) , and H (·) are three individual neural networks, as shown by the

networks

210, 220, and 230 in FIG. 2. F ₁ (·) and F ₂ (·) are encoders for transformation parameters θ with a dimension of m and images x with a dimension of n respectively, and their encodings are added together in the semantic space and input into the semantic layer H (·) . Generally, a U-Net model may be used for the surrogate neural network 200. In some embodiments, all BatchNorm layers of the U-Net model may be replaced with GroupNorm layers, since the surrogate neural network 200 may be used in low batchsize setting. In other embodiments, the U-Net model may also be replaced by other networks used in image segmentation or superresolution such as Res-UNet or EDSR (Enhanced Deep Residual Network) . Specifically, the U-Net, Res-UNet or EDSR network may be adopted for the neural network H (·) , and several simple convolutional or linear layers may be adopted for F ₁ (·) and F ₂ (·) . All of these neural networks may be trained by using an Adam optimizer with an initial learning rate of 0.001 that decays every 50 epochs until convergence, for example. L1-loss may be used to train the surrogate neural network, which may achieve better accuracy than others.

The surrogate neural network may be much easier to analyze and can be certified by introducing a dimension augmentation strategy for both transformation parameters and input images. As shown in Fig. 2, by introducing an augmented noise

with a dimension of d in the layers of the surrogate neural network 200, randomized smoothing can be extended to handle these complex semantic transformations. In this way, the complex semantic transformations can be viewed as the superposition of a resolvable part and a non-resolvable residual part in the augmented semantic space. Then, the augmented noise may be used to control the non-resolvable residual part of the augmented dimension d ≥ m + n. The augmentation for noise is from

to

By certifying the semantic transformation using the surrogate neural network, it is possible to certify the original transformation if the approximation error is within an acceptable region. This method is flexible and scalable because the surrogate neural network has a uniform form for analysis and may be trained automatically.

In one embodiment, to keep the dimension consistent, the input data x may also be augmented to

by padding 0 entries, where d = m + n. Accordingly, the augmented data

and the augmented parameter

may be defined as

where the additional parameters

are sampled from g′ (θ′) , and the joint distribution of θ′and θ is

where

Then, the generalized smoothed classifier may be defined as

where

is the “augmented target classifier” that is equivalent to the original classifier f when constrained on the original input x, which means

Note that now all the functions are augmented for a d-dimensional input. Then, the surrogate neural network may be augmented to represent the augmented transformation

as

where

and

are parts of the augmented surrogate neural network. By carefully designing the interaction between the augmented parameters and the original parameters, the augmented transformation may be turned to a resolvable semantic transformation. It does not change the original surrogate neural network when constraining to the original input x and θ. Specifically,

and

may be designed as follows:

As shown in FIG. 2, with the input image x and transformation parameter θ, the surrogate neural network 200 may output augmented noisy images 240, by adding augmented transformation parameter

into the surrogate neural network 200. Next, the augmented noisy images 240 may be input into the targe base classifier 250. Then, a certified robust region 260 with a radius R may be calculated based on the generalized randomized smoothing method.

In order to analysis the certified bound for non-resolvable transformations by using the surrogated neural network, the notations may be simplified as follows:

Then, it (referred as theorem 2 herein) can be proved that: for any classifer f (x) with a corresponding randomized smoothed classifer G (x) defined in equation (10) , if there exists p _A and

satisfying

then

holds for any ||ξ|| ₂ ≤ R, where

and the coefficient M*is defined as

From equations (14) - (15) it can be seen that the certified radius is similar to the result in theorem 1 as described above. Compared with resolvable transformations, a new type of augmented noise needs to be introduced when constructing the GRS classifier. This isotropic noise may have the same dimension as the data and may be added to the intermediate layers of surrogate neural networks. The purpose of the augmented noise is to construct a closed subspace using additional dimensions. In the augmented space, the Jacobian matrix of semantic transformations becomes invertible. The coefficient M*depends on the norm of the difference of two Jacobian matrices and is independent of the target classifier.

In one embodiment, a linear transformation may be adopted as F ₁ (θ) , i.e.,

F ₁(θ) = A ₁θ + b ₁, (16)

where

does not sacrifice the precision of the surrogate neural network. Thus,

After substituting the item in equation (15) , it is only needed to optimize ξ to calculate M*which can make the bound tighter. Additionaly, two Gaussian distributions may be used as the original noise distribution and the distribution of the additional noise part of the augmented noise, i.e.,

and

If there exists p _A and

satisfying

then

holds for any ||ξ||2 ≤ R, where

where Ψ (·) is the inverse CDF of the standard Gaussian distribution, and the coefficient M*is defined as

From equation (18) it can be seen that M*is influenced by two factors. One is the standard deviation of the two noise distributions. The other is the norm of the Jacobian matrix

It can be viewed as the residual of the non-resolvable part of the transformation. Accordingly, the various semantic transformations may be decomposed into a resolvable part and a residual part. The non-resolvable residual part may be handled by introducing an additional noise with standard deviation σ2.

Although there may exist approximation error between the surrogate neural network and the original real semantic transformation, it can be proved that if the simulation of the semantic transformation has a small enough error, i.e.,

where

is the real semantic, then there exists a constant ratio

which may not depend on the target classifier, and the certified radius for the real semantic transformation satisfies

R _r＞R (1-Aε) , (19)

where R may be the certified radius in equations (14) or (17) for the surrogate neural network.

From equation (19) it can be seen that the reduction of the certified radius may be influenced by two factors. The first one is the approximation error ∈ between the surrogate transformation and the real semantic transformation. The second one is the ratio A about the norm of the Jacobian matrix for some layers of the surrogate neural network. This is also an inherent property of the semantic transformation itself and may not depend on the target classifier.

FIG. 3 illustrates a flow chart of a method 300 for certifying defense against image transformation in accordance with one aspect of the present disclosure. In block 310, the method 300 may comprise generating a surrogate neural network consisting of three individual neural networks to simulate the image transformation. The three individual neural networks includes: a first neural network which is an encoder for transformation parameters, a second neural network which is an encoder for input images, and a third neural network which is a transformation network for outputs of the first and second neural networks. The transformation parameters may be augmented in dimension and added to the surrogate neural network.

In some embodiments, the transformation parameters are augmented to a dimension equal to or greater than a sum of a dimension of the input images and a dimension of the transformation parameters. As shown in equation (9) , the augmented transformation parameters may comprise the transformation parameters and additional transformation parameters. The transformation parameters and the additional transformation parameters may be sampled from two Gaussian distributions. The additional transformation parameters may be sampled to turn the image transformation to a resolvabe semantic transformation. To keep the dimension consistent, the input image may also be augmented by padding 0 entries. The surrogate neural network may be further augmented to represent the augmented transformations.

The surrogate neural network may use a U-Net model, wherein all BatchNorm layers of the U-Net model may be replaced with GroupNorm layers, while the surrogate neural network is used in low batchsize setting. The surrogate neural network may also use other networks used in image segmentation or superresolution, such as, Res-UNet or EDSR (Enhanced Deep Residual Network) . Specifically, the first and second neural networks in the surrogate neural network may comprise several simple convolutional or linear layers, and the third neural network may be based on a U-Net, Res-UNet or EDSR network. All these neural networks may be trained based on L1-loss and by using Adam optimizer.

In block 320, the method 300 may comprise certifying the defense of a target deep learning model against the image transformation by using the surrogate neural network based on randomized smoothing. In one embodiment, a certified radius of a robust region of a base classifier for the image transformations may be calculated based on the functions of the surrogate neural network as shown in equations (14) - (15) and (17) - (18) . Further analysis on the properties of semantic transformations may be performed by using the surrogate neural network.

FIG. 4 illustrates a block diagram of an apparatus 400 for certifying defense against image transformation in accordance with one aspect of the present disclosure. The apparatus 400 may comprise a memory 410 and at least one processor 420. The processor 420 may be coupled to the memory 410 and configured to perform the method 300 described above with reference to FIG. 3. The processor 420 may be a general-purpose processor, or may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. The memory 410 may store the input data, output data, the surrogate neural network generated by processor 420, and/or instructions executed by processor 420.

In one embodiment, the processor 420 may be configured to generate a surrogate neural network consisting of three individual neural networks for simulating the image transformation. A first neural network of the three individual neural networks is an encoder for transformation parameters. A second neural network is an encoder for input images. And a third neural network is a transformation network for outputs of the first and second neural networks. The transformation parameters are augmented in dimension and added to the surrogate neural network. The processor 420 may be further configured to certify the defense against the image transformation by using the surrogate neural network based on randomized smoothing.

The various operations, modules, and networks described in connection with the disclosure herein may be implemented in hardware, software executed by a processor, firmware, or any combination thereof. According an embodiment of the disclosure, a computer program product for computer vision processing may comprise processor executable computer code for performing the method 300 described above with reference to FIG. 3. According to another embodiment of the disclosure, a computer readable medium may store computer code for computer vision processing, the computer code when executed by a processor may cause the processor to perform the method 300 described above with reference to FIG. 3. Computer-readable media includes both non-transitory computer storage media and communication media including any medium that facilitates transfer of a computer program from one place to another. Any connection may be properly termed as a computer-readable medium. Other embodiments and implementations are within the scope of the disclosure.

The preceding description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the various embodiments. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the scope of the various embodiments. Thus, the claims are not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the following claims and the principles and novel features disclosed herein.

Claims

A method for certifying defense against image transformation, comprising:

generating a surrogate neural network consisting of three individual neural networks to simulate the image transformation, wherein a first neural network is an encoder for transformation parameters, a second neural network is an encoder for input images, a third neural network is a transformation network for outputs of the first and second neural networks, and the transformation parameters are augmented in dimension and added to the surrogate neural network; and

certifying the defense against the image transformation by using the surrogate neural network based on randomized smoothing.
The method of claim 1, wherein the transformation parameters are augmented to a dimension equal to or greater than a sum of a dimension of the input images and a dimension of the transformation parameters.
The method of claim 1, wherein the augmented transformation parameters comprises the transformation parameters and additional transformation parameters, and wherein the transformation parameters and the additional transformation parameters are two Gaussian distributions.
The method of claim 3, wherein the additional transformation parameters are sampled to turn the image transformation to a resolvabe transformation.
The method of claim 1, wherein the first and second neural networks comprise convolutional or linear layers, and the third neural network is based on a U-Net, Res-UNet or EDSR network.
The method of claim 1, wherein the generating the surrogate neural network comprises training the surrogate neural network based on L1-loss.
The method of claim 1, wherein the input images comprises traffic images for assisting autonomous driving.
An apparatus for certifying defense against image transformation, comprising:

a surrogate neural network consisting of three individual neural networks for simulating the image transformation, wherein a first neural network is an encoder for transformation parameters, a second neural network is an encoder for input images, a third neural network is a transformation network for outputs of the first and second neural networks, and the transformation parameters are augmented in dimension and added to the surrogate neural network, and

wherein the surrogate neural network is used to certify the defense against the image transformation based on randomized smoothing.
The apparatus of claim 8, wherein the transformation parameters are augmented to a dimension equal to or greater than a sum of a dimension of the input images and a dimension of the transformation parameters.
The apparatus of claim 8, wherein the augmented transformation parameters comprises the transformation parameters and additional transformation parameters, and wherein the transformation parameters and the additional transformation parameters are two Gaussian distributions.
The apparatus of claim 10, wherein the additional transformation parameters are sampled to turn the image transformation to a resolvabe transformation.
The apparatus of claim 8, wherein the first and second neural networks comprise convolutional or linear layers, and the third neural network is based on a U-Net, Res-UNet or EDSR network.
The apparatus of claim 8, wherein the surrogate neural network is trained based on L1-loss.
The apparatus of claim 8, wherein the input images comprises traffic images for assisting autonomous driving.
An apparatus for certifying defense against image transformation, comprising:

a memory; and

at least one processor coupled to the memory and configured to perform the method of one of claims 1-7.
A computer readable medium, storing computer code for certifying defense against image transformation, the computer code when executed by a processor, causing the processor to perform the method of one of claims 1-7.
A computer program product for certifying defense against image transformation, comprising: processor executable computer code for performing the method of one of claims 1-7.