CN114170443A - Image semantic anti-attack sample generation system and method based on image brightness mapping - Google Patents

Image semantic anti-attack sample generation system and method based on image brightness mapping Download PDF

Info

Publication number
CN114170443A
CN114170443A CN202111434878.2A CN202111434878A CN114170443A CN 114170443 A CN114170443 A CN 114170443A CN 202111434878 A CN202111434878 A CN 202111434878A CN 114170443 A CN114170443 A CN 114170443A
Authority
CN
China
Prior art keywords
image
mapping
original
brightness
matrix
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202111434878.2A
Other languages
Chinese (zh)
Inventor
孙亮儒
黄怿豪
诸嘉逸
缪炜恺
蒲戈光
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
East China Normal University
Original Assignee
East China Normal University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by East China Normal University filed Critical East China Normal University
Priority to CN202111434878.2A priority Critical patent/CN114170443A/en
Publication of CN114170443A publication Critical patent/CN114170443A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Artificial Intelligence (AREA)
  • General Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Evolutionary Computation (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Computational Linguistics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Health & Medical Sciences (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Evolutionary Biology (AREA)
  • General Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Image Analysis (AREA)

Abstract

The invention discloses an image semantic anti-attack sample generation system based on image brightness mapping, which comprises the following steps: the image reading module is used for reading original image information and dividing the color and the brightness of an image to obtain an image brightness matrix and an image color matrix; the image mapping module is used for intercepting an effective brightness matrix and an image invalid part according to the brightness matrix of the original image, and mapping the effective brightness matrix by using a mapping function to obtain an original attack sample; the target network module is used for putting the original attack sample into a target network for reasoning, reversely propagating after a reasoning result is obtained, and adjusting a mapping function to obtain a new mapping function; and the image evaluation module is used for skipping, saving or updating the current original attack sample according to the image reasoning result and the image quality score. The invention also discloses a method for generating the attack sample by using the attack sample generation system.

Description

Image semantic anti-attack sample generation system and method based on image brightness mapping
Technical Field
The invention belongs to the field of artificial intelligence security, and relates to an image semantic anti-attack sample generation system and method based on image brightness mapping.
Background
Deep Neural Networks (DNNs) are vulnerable to targeted attacks against samples. Wherein a portion of the challenge sample is perturbed by LpNorm limits to ensure imperceptibility. However, these attack examples are not natural where the square error is low (e.g., sky, walls, etc.). Other attacks do not limit the perturbation to obtain better robustness and transferability, so these antagonistic examples can be used for transfer-based black-box attacks. But these methods discard a large part of the naturalness.
Disclosure of Invention
In order to solve the defects of the prior art, the invention aims to provide an image semantic anti-attack system based on image brightness mapping, which can be used for carrying out non-restriction (such as L)pNorm) while preserving naturalness; by generating a natural attack image for a target network according to an input RGB image, the classifier can be enabled to generate errors under the condition that human eyes cannot recognize the images.
Specifically, the invention provides an image semantic anti-attack sample generation system based on image brightness mapping, which comprises:
the image reading module is used for reading original image information and dividing the color and the brightness of the original image to obtain an image brightness matrix and an image color matrix;
preferably, the input image format of the image reading module is Tensor, and the Tensor in the LAB color space is obtained by converting the RGB space into the LAB space image before the brightness and the color of the original image are divided.
The image mapping module is used for intercepting an effective brightness matrix and an image invalid part according to the brightness matrix of the original image, mapping the effective brightness matrix by using a mapping function, and integrating the effective brightness matrix and the image invalid part after mapping is finished to obtain an original attack sample;
preferably, the image mapping module comprises:
the mapping and dividing unit is used for dividing an effective brightness matrix and an image invalid part based on an image brightness effective value according to an original image brightness interval; specifically, the effective brightness range of the image is from the minimum value to the maximum value of the image in the brightness space.
The image mapping unit is used for initializing a mapping function, mapping the effective brightness matrix according to the mapping function, and integrating the mapped effective brightness matrix and the image invalid part obtained in the mapping division unit to obtain an original attack sample;
the target network module is used for putting the original attack sample into a target network for reasoning, reversely propagating after a reasoning result is obtained, and adjusting a mapping function to obtain a new mapping function;
preferably, the target network module comprises:
the image reasoning unit is used for reasoning the original attack sample in the image reasoning unit to obtain a reasoning result, and if the image is an image classification task, classifying the image and judging the category of the image;
a regularization unit for making the loss value contain a term whose mapping function slope is not 0.
Preferably, the loss function after adding the regularization term is:
Loss＝LC&W(x，l)-ηsum(|θ|)/K
wherein L isC&W(. is) C&W loss, namely subtracting a maximum value except the class from a value corresponding to the class of the label l in a reasoning result obtained by the image reasoning unit of the input image x, wherein sum (| theta |)/K is a regularization term; η is a regularization rate used to adjust the degree of regularization, i.e., the proportion of the regularization term in the loss function, and in the present invention, η is 0.5.
The back propagation unit is used for correcting the mapping function according to the obtained inference result and the value of the regularization term obtained by the regularization unit and by combining the learning rate; obtaining a partial derivative value of the mapping function to the inference result according to the inference result, the regularization item and the network model, and adjusting the mapping function by combining the learning rate to obtain a new mapping function;
preferably, the learning rate is determined by experimental results; the learning rate used by the invention is 0.3;
preferably, the direction of parameter modification, i.e. increasing or decreasing the parameter value, is determined according to the partial derivative value, i.e. the degree of influence of the parameter of the mapping function on the final result; in particular, the partial derivatives can be calculated directly;
preferably, after the direction of parameter adjustment is determined by the partial derivative value, the adjustment size of each step is the learning rate; in particular, the parameter θ is obtained if a certain calculation is performedkIs positive, the learning rate is 0.3, the parameter theta after the iteration iskAdding 0.3 to the value of (a);
and the image evaluation module is used for determining to skip or save or update the current image according to the image inference result and the image quality score.
Preferably, the image evaluation module comprises:
the image quality evaluation unit is used for obtaining the image quality score of the generated original attack sample through an image evaluation algorithm;
and the image decision unit skips, saves or updates the current image according to the inference result obtained by the image inference unit and the image quality score obtained by the image quality evaluation unit.
In the image mapping module, the slopes of the segments represented by the mapping function are preferably initialized randomly within the original image luminance range.
Preferably, the mapping function is initialized randomly, that is, the mapping function uses a random value, and compared with the mapping function which is initially 1.0, the mapping function can better exert the capability of attacking the system and obtain higher attack success rate and image quality;
preferably, in the invention, the parameter value of the mapping function is set to be a random floating point number of 0.8-1.2;
preferably, the image mapping unit uniformly segments the luminance according to the mapping function, and the luminance within each segment changes in value with the same slope according to the corresponding parameter.
In the target network module, preferably, the image inference unit is a convolutional neural network, including but not limited to: LeNet, ResNet, DenseNet.
Preferably, the regularization term is the inverse of the absolute value of the values of the mapping function averaged.
In the image evaluation module, preferably, the image quality evaluation unit obtains an image quality score using LPIPS as a scoring criterion.
Preferably, LPIPS is a judgment index that mimics human visual perception for measuring the degree of similarity of images in a manner consistent with human judgment.
Preferably, a lower LPIPS value indicates better image quality, and an image quality score, i.e., the inverse of the LPIPS value, is higher.
Preferably, the image decision unit compares the image quality score of the successfully attacked image obtained by the image quality evaluation unit with the image quality score of the attack sample stored in the previous iteration, and selects the image with higher image quality score for storage; and if the current image attack is unsuccessful, saving the current image as a failed image. In the image classification task, the successful attack means that the type of the attacked image is different from the type of the attacked image obtained by reasoning through the image reasoning module; the unsuccessful attack means that the type of the attacked image obtained by reasoning the image reasoning module is the same as the label.
The invention also provides an image semantic anti-attack image generation method based on image brightness mapping, which comprises the following steps:
inputting an original image into an image reading module, converting the original image into a Tensor format, and converting an RGB color space into an LAB color space to obtain a brightness matrix and a color matrix of the input image;
step two, an image mapping module intercepts an effective brightness matrix from the original image brightness matrix obtained in the step one, maps the effective brightness matrix by using a mapping function, and integrates the effective brightness matrix with an image invalid part after mapping is finished to obtain an original attack sample;
inputting an original attack sample into a target network module, carrying out reasoning by using an image reasoning unit and obtaining a reasoning result, and modifying a mapping function by a back propagation unit according to a classification result obtained by the reasoning and a regularization item value obtained by the regularization unit in combination with a learning rate;
step four, the image evaluation module skips, saves or updates the original attack sample obtained in the step two by selecting the inference result in the step three and the image quality score obtained by the image quality evaluation module;
and step five, repeating and iterating the steps two to four to obtain a final attack sample.
In the first step, after the image reading module reads the image through an OpenCV (open content library) of python, a Numpy library is used for adjusting the image format from BGR to RGB, and then the brightness information and the color information of the image are obtained through a conversion formula of an RGB space and an LAB space; converting the original picture into a Tensor format and performing through a torch library of python; the formula for converting the data in the Tensor format from the RGB color space to the LAB color space is as follows:
Figure BDA0003381276320000041
Figure BDA0003381276320000042
Figure BDA0003381276320000043
Figure BDA0003381276320000044
Figure BDA0003381276320000045
wherein, R, G and B respectively represent three channel components corresponding to the original picture in RGB color space, and the three channel components are two-dimensional matrixes with the same size as the picture; x, Y, Z are the three channel components of the representation on the XYZ color space resulting from the representation of the image on the RGB color space; xn，Yn，ZnIs a fixed value, usually Xn＝0.95047，Yn＝1.0，Zn1.08883; by matrix operations on the XYZ color space, a representation of the original image on the LAB color space can be obtained.
In the second step, the setting of the effective brightness is determined according to the maximum value and the minimum value of the input picture in the brightness space; in the luminance space, the part of the luminance matrix with the matrix value larger than 0 after subtracting the minimum value is the effective luminance of the input picture; after the interception of the effective brightness matrix is finished, simultaneously saving the invalid part of the image;
the mapping function is as follows:
Figure BDA0003381276320000046
Figure BDA0003381276320000047
wherein K represents the number of divided stages, θkDenotes the parameter, x, corresponding to the k-th segmentkBrightness value, F, representing an arbitrary position in the original imageθ(xk) And mapping the corresponding position of the image to obtain a new brightness value.
In the third step, the image inference unit is a convolutional neural network, and generally receives the input of (b,3, W, H), extracts features through a plurality of convolutional layers and a plurality of pooling layers, and then connects a full connection layer with a classification number c for classification, wherein b is the batch processing number, 3 represents RGB three-dimension, W is the image width, and H is the image height; in the image classification task, after one or b pictures read into a Tensor format are input into a target network, a 1-dimensional vector representing the class of the pictures judged by the target network is obtained. The classification number of the 1-dimensional vector is c parameters which respectively represent the possibility of the class to which the picture belongs by the target network inference, wherein the class with the highest numerical value is the class of the picture obtained by the target network inference;
specifically, the network structure of LeNet is shown in fig. 5; after extracting features of an input picture through a plurality of convolutional layers such as C1 and C3 and a plurality of pooling layers (down-sampling layers) such as S2 and S4, connecting a plurality of full-connection layers (usually one layer) with upper figures of C5, F6 and OUTPUT, wherein the parameter number of the OUTPUT layers is the target classification number, for example, in the case of handwritten number recognition shown in FIG. 5, 10 numbers including 0 to 9 are used, so that the OUTPUT layers have 10 parameters; the remaining ResNet and DenseNet are similar to the network characteristics;
and in the fourth step, for the attack image generated by iteration of a certain time, the generated image is put into the image reasoning module in the third step, whether the generated image can cause the image reasoning module to be wrongly classified is checked, and if the generated image is wrongly classified, the attack is successful. And comparing the score of the image which is successfully attacked in the image evaluation module with the currently stored image, namely the corresponding score of the attack sample stored in the previous iteration, if the score is higher, storing the new image, and if not, discarding the new image and entering the next iteration.
The beneficial effects of the invention include: pioneering attacks on the brightness of the image; based on the characteristics of brightness, a targeted regular optimization is innovatively provided, and the attack accuracy and the image quality are obviously improved under the optimization; on the basis of a plurality of image related tasks and a plurality of image data sets, the attack of the method has excellent attack success rate, and meanwhile, the method is obviously superior to the existing unlimited attack method on a plurality of commonly used image quality evaluation indexes. By the image brightness attack sample generation system based on the conductible mapping, the attack sample with excellent image quality can be obtained with higher attack success rate, so that the classification of the target model is wrong under the condition that human eyes look natural.
Drawings
FIG. 1 is a schematic diagram of an image semantic anti-attack system based on image brightness mapping according to the present invention;
FIG. 2 is a schematic structural diagram of an image semantic anti-attack system based on image brightness mapping according to the present invention;
FIG. 3 is a mapping example diagram of an image semantic anti-attack system based on image brightness mapping according to the present invention;
FIG. 4 is a flowchart of a method for generating image semantic anti-attack samples based on image brightness mapping according to the present invention.
Fig. 5 is a schematic diagram of a convolutional neural network structure in the image inference unit 1 according to the present invention.
Fig. 6 is a schematic diagram of an original image, a picture generated by the existing PGD method, and a picture generated by the OLF method in the embodiment of the present invention.
Detailed Description
The invention is further described in detail with reference to the following specific examples and the accompanying drawings. The procedures, conditions, experimental methods and the like for carrying out the present invention are general knowledge and common general knowledge in the art except for the contents specifically mentioned below, and the present invention is not particularly limited.
According to an embodiment of the present application, an image semantic anti-attack sample generation system based on image brightness mapping is provided, as shown in fig. 1, including:
the image reading module is used for converting an input original RGB image into an LAB space image and dividing the color and the brightness of the image to obtain an image brightness matrix and an image color matrix;
the image mapping module is used for intercepting an effective brightness matrix and an image invalid part according to the brightness matrix of the original image, mapping the effective brightness matrix by using a mapping function, and integrating the effective brightness matrix and the image invalid part after mapping is finished to obtain an original attack sample;
the target network module is used for putting the original attack sample into a target network for reasoning, reversely propagating after a reasoning result is obtained, and adjusting a mapping function to obtain a new mapping function;
and the image evaluation module is used for determining to skip or save or update the current image according to the image reasoning result and the image quality evaluation result.
As shown in fig. 2, the image mapping module includes:
the mapping and dividing unit is used for dividing an effective brightness matrix and an image invalid part based on an image brightness effective value according to an original image brightness interval;
the image mapping unit is used for initializing a mapping function; in the invention, the parameter value of the mapping function is set to be a random floating point number of-0.2-0.8, so that the capability of attacking the system can be better exerted than the mapping function which is initially all 1.0, and higher attack success rate and image quality are obtained; mapping the effective brightness matrix according to a mapping function, and integrating the mapped effective brightness matrix with an image invalid part obtained in the segmentation unit to obtain an original attack sample;
as shown in fig. 2, the target network module includes:
the image reasoning unit is used for putting the original attack sample into a target network for reasoning to obtain a reasoning result;
a regularization unit for making the loss value contain a term whose mapping function slope is not 0.
The back propagation unit is used for obtaining a partial derivative value of the mapping function to the inference result according to the inference result, the regularization item and the network model and adjusting the mapping function by combining the learning rate to obtain a new mapping function;
as shown in fig. 2, the image evaluation module includes:
the image quality evaluation unit obtains an image quality score of the generated image through an image evaluation algorithm;
and the image decision unit skips, saves or updates the current image according to the inference result obtained by the image inference unit and the image quality score obtained by the image quality evaluation module.
The slope of the partition represented by the mapping function is randomly initialized within the luminance range of the original image.
The image mapping unit segments the luminance according to a mapping function, and the luminance changes in each segment according to the same slope.
The image inference unit is a convolutional neural network, including but not limited to: LeNet, ResNet, DenseNet.
The regularization term is the inverse of the mean of the absolute values of the mapping function.
The image quality evaluation unit uses the reciprocal of LPIPS as a scoring standard to obtain an image quality score; the higher the image quality, the lower the LPIPS value, and the higher the image quality score.
The image decision unit compares the image quality score of successful attack with the image quality score of the attack sample stored in the previous iteration, and selects the storage with higher score; and if the current image attack is unsuccessful, entering next iteration.
For a better understanding, the workflow of the present invention will be described below.
As shown in fig. 2, the image reading module first performs color space conversion on an input RGB image to obtain a representation of the image in an LAB space, and then divides colors and luminances to obtain an image luminance matrix and an image color matrix.
Preferably, the input image format is Tensor, and the conversion from RGB space to LAB space image results in Tensor in LAB color space.
And a mapping dividing unit in the image mapping module obtains the effective value range of the image brightness according to the brightness matrix, extracts the effective value of the image brightness matrix, obtains the effective brightness matrix and stores the ineffective part of the image.
In the image mapping unit, each parameter of the mapping function represents a slope when the image is mapped to generate an attack sample. And the effective brightness matrix is subjected to mapping function transformation to obtain a new effective brightness matrix, and then the effective brightness matrix is integrated according to invalid numerical value information divided by the mapping dividing unit to obtain a complete attack sample.
And the image reasoning unit in the target network module performs reasoning and classification according to the generated attack sample to obtain a classification result, namely, the image is judged to be a specific object, such as a vehicle, a person, a fish and the like. And the regularization unit obtains the value of the regularization item according to the parameters of the mapping function. And the back propagation unit is used for obtaining the trend of modifying the parameters of the mapping function by derivation according to the classification result and the regularization item, and modifying the mapping function by combining the learning rate.
Preferably, the regularization term is the inverse of the mean of the absolute values of the mapping function parameters, so that as few parameters as 0 exist in the mapping function.
The image mapping module is described in further detail below.
And the mapping dividing unit in the image mapping module obtains the effective range of the brightness matrix according to the maximum and minimum values of the brightness matrix obtained by the image reading module in the brightness space.
Assume that the image effective brightness ranges from 0.25-0.75 as shown in fig. 3.
And according to the effective range, the mapping and dividing unit cuts the brightness matrix to obtain an effective brightness matrix.
Specifically, after the minimum value of the entire luminance matrix is stored, the minimum value is subtracted, and then the portion of the new matrix with the median value larger than 0 (and smaller than 0.5) is the effective luminance, and the new matrix is the effective luminance matrix.
An image mapping unit in the image mapping module initializes a mapping function.
The mapping function is as follows:
Figure BDA0003381276320000071
Figure BDA0003381276320000072
in particular, the mapping function is a one-dimensional vector with K parameters, i.e. K floating-point numbers. Wherein, K represents the number of segments for dividing the effective brightness, and each parameter value represents the relative slope shared by each segment of brightness after division.
As shown in fig. 3, for a picture with an effective luminance of 0.25 to 0.75, let K be 4, the parameters of the mapping function are represented as a portion from x being 0.25 to x being 0.75 in the image, where the parameters of the mapping function represent the relative slope of each segment.
Specifically, the first segment of the effective matrix is mapped first. 0 to 0.5 are divided into 4 segments, and the horizontal axis of each segment is 0.125.
Multiplying the part of the effective brightness matrix which is more than or equal to 0 and less than 0.125 by a first parameter to obtain a matrix M after corresponding position mapping1I.e. Fθ(xk)＝xk·θ1，xkE [0, 0.125); subtracting 0.125 from the effective brightness matrix, multiplying the part of the new matrix with the subtracted 0.125, which is greater than or equal to 0 and less than 0.125, by a second parameter, and adding M obtained by the previous calculation1Obtaining M2I.e. by
Figure BDA0003381276320000081
Until the 4 parts are mapped, a new effective brightness matrix M is obtained4. Then, a new effective brightness matrix M is obtained4Scaling to 0 or more and 0.5 or less in the same ratio to obtain a new effective luminance matrix.
In particular, due to the characteristics of brightness, the mapping function performs slope-independent mapping on the effective brightness of the image, but the value range after mapping is still within the effective range of the original image.
The image mapping unit maps the effective brightness matrix through a mapping function to obtain an attacked effective brightness matrix, and supplements the effective brightness matrix to an image invalid part, namely, each parameter of the effective brightness matrix is added with the minimum value of the original brightness to supplement the attack brightness matrix.
And according to the image color matrix stored when the original image is segmented by the image reading module, combining the attack brightness matrix and the image color matrix to obtain an original attack sample.
Specifically, the mapping function is directed to the mapping of the effective luminance, and the region (minimum value) other than the effective luminance in fig. 3 is stored when the mapping dividing unit cuts the luminance matrix.
Taking an arbitrary image as an example, the following algorithm will be described in further detail.
The algorithm of the image semantic anti-attack sample generation method based on image brightness mapping is as follows:
Figure BDA0003381276320000091
when the program runs, an original image x and a label l of the image need to be input.
Iteration number I, learning rate alpha, regularization rate eta and mapping function FθThe three parameters of the number K of segments to be divided are also freely set by the user.
Image x first enters the image reading module. The image reading module reads the pictures in the picture formats of png/jpg and the like through an OpenCV (open CV) library of python to obtain a Numpy array expressed by BGR (pixel value matching) and converts the array dimension into RGB (red, green and blue) through the Numpy library, and the values in the array represent the pixel values of the original pictures. And then, the pictures are converted into a Tensor format through the torreh library, so that the subsequent processing efficiency is obviously improved. And converting the Tensor expressed by the RGB color space into the Tensor in the LAB color space according to a formula for converting the RGB color space into the LAB color space.
The input image x represented in the LAB color space contains three dimensions, i.e. brightness
Figure BDA0003381276320000092
And color
Figure BDA0003381276320000093
Using LAB colour space by image reading module
Figure BDA0003381276320000094
Matrix is divided into
Figure BDA0003381276320000095
And
Figure BDA0003381276320000096
wherein
Figure BDA0003381276320000097
I.e. the luminance matrix of x.
The image dividing unit in the image mapping module is based on the brightness matrix
Figure BDA0003381276320000098
Value range of (a), dividing the effective luminance matrix x*。
An image mapping unit in the image mapping module initializes the mapping function according to the set k value
Figure BDA0003381276320000099
The mapping function is a one-dimensional vector with K parameters. Wherein, K represents the number of segments for dividing the effective brightness, each parameter value represents the slope shared by each segment of the divided brightness, and each parameter is a one-dimensional vector of a random floating point number of 0.8-1.2
Figure BDA0003381276320000101
An image mapping unit in the image mapping module divides the effective brightness matrix into equidistant K segments, and each segment is multiplied by a mapping function
Figure BDA0003381276320000102
And obtaining the effective brightness matrix after attack according to the slope parameters. And then, according to the range of the original effective brightness, zooming the attacked effective brightness matrix to ensure that the value range of the attacked effective brightness matrix is the same as that of the original effective brightness, and obtaining a new effective brightness matrix x* 0。
Dividing x according to data stored when effective brightness matrix is divided by image dividing unit* 0Reverting to a complete luminance matrix
Figure BDA0003381276320000103
Divided by an image reading moduleCut out
Figure BDA0003381276320000104
The matrix being a luminance matrix
Figure BDA0003381276320000105
Adding color part to generate attack image
Figure BDA0003381276320000106
Figure BDA0003381276320000107
Attack image x'0The image mapping module generates the image, and transmits the image to the image inference unit in the target network module for processing.
The image inference unit is the target network of attacks, such as ResNet50 for handling image classification problems. After the image passes through the ResNet50, a one-dimensional vector logit containing a plurality of parameters of the classification is obtained at the penultimate layer according to the probability that the image belongs to each classification.
According to the input image label l, corresponding classification in the logit, namely the probability of the l class is subtracted by the maximum value except the probability of the l class to obtain C&W loss LC&W(x′i-1，l)。
In particular, since the mapping function FθWhen the value of (1) is 0, the image may be unnatural, and therefore, the regularization term sum (| θ |)/K is added so that F is equal toθThe smaller the absolute value of the parameter(s) of (a), the greater the overall loss, so that finally FθThe probability of a value of 0 decreases; and the degree of regularization is adjusted by a hyper-parametric regularization rate η, where η is 0.5 in the present invention.
For the routing L in the target networkC&WComplex-loss function in combination with regularization term relating to a mapping function FθGradient g is obtained.
According to the gradient g and the learning rate alpha, for
Figure BDA0003381276320000108
Parameter theta of0Make a modificationTo obtain
Figure BDA0003381276320000109
By passing
Figure BDA00033812763200001010
Modifying the effective luminance x of an original image*To obtain a new effective brightness x* 1And then obtaining a new attack sample x 'by reduction'1And then circularly iterating the above steps until the iteration number reaches I, namely obtaining the sample x ' ═ x ' after the attack 'i。
And performing iteration I-round attack on each picture needing attack. After iteration is finished, if a sample with successful attack exists, the picture is successful in attack, and if not, the picture is failed in attack.
As shown in fig. 6, taking the image of the bird as an example, both the PGD and OLF attack methods generate an anti-attack sample that attacks successfully, i.e., both the target networks are determined to be not the bird. Under the condition of comparing with the original image, the attack image generated by the method is obviously closest to the original image; if the original image is not seen, the image generated by the method can be observed to be most natural; PGD is classical LpThe norm limits the attack, and the background of the picture generated by the method can be seen to obviously see noise.
Particularly, as for the index for determining the naturalness of the picture, LPIPS is one of the most commonly used indexes. LPIPS is a judgment index that simulates human visual perception and measures the degree of similarity of images in a manner that conforms to human judgment. The lower the LPIPS value, the better the image quality. Therefore, the image quality evaluation unit in the image evaluation module of the invention uses LPIPS as an evaluation index, and the results obtained on the common data set are superior to the prior method.
Specifically, LPIPS of the picture shown in fig. 6 are respectively: 0.1480(PGD) and 0.0224 (OLF).
The protection of the present invention is not limited to the above embodiments. Variations and advantages that may occur to those skilled in the art may be incorporated into the invention without departing from the spirit and scope of the inventive concept, and the scope of the appended claims is intended to be protected.

Claims (12)

1. An image semantic anti-attack sample generation system based on image brightness mapping is characterized by comprising:
the image reading module is used for reading original image information, converting an input RGB image into an LAB space image, and segmenting the color and the brightness of the image to obtain an image brightness matrix and an image color matrix;
the image mapping module is used for intercepting an effective brightness matrix and an image invalid part according to the brightness matrix of the original image, mapping the effective brightness matrix by using a mapping function, and integrating the effective brightness matrix and the image invalid part after mapping is finished to obtain an original attack sample;
the target network module is used for putting the original attack sample into a target network for reasoning, reversely propagating after a reasoning result is obtained, and adjusting a mapping function to obtain a new mapping function;
and the image evaluation module is used for skipping, saving or updating the current original attack sample according to the image reasoning result and the image quality score.
2. The generation system of claim 1, wherein the image mapping module comprises:
the mapping and dividing unit is used for dividing an effective brightness matrix and an image invalid part based on an image brightness effective value according to an original image brightness interval;
and the image mapping unit is used for initializing a mapping function, mapping the effective brightness matrix according to the mapping function, and integrating the mapped effective brightness matrix and the image invalid part obtained in the mapping division unit to obtain an original attack sample.
3. The generation system of claim 1, wherein the target network module comprises:
the image reasoning unit is used for putting the original attack sample into the image reasoning unit for reasoning to obtain a reasoning result;
a regularization unit for making the loss value contain a term whose mapping function slope is not 0;
and the back propagation unit is used for obtaining the partial derivative value of the mapping function to the inference result according to the obtained inference result and the value of the regularization term obtained by the regularization unit and adjusting the mapping function by combining the learning rate to obtain a new mapping function.
4. The generation system of claim 1, wherein the image evaluation module comprises:
the image quality evaluation unit obtains an image quality score of the generated image through an image evaluation algorithm;
and the image decision unit skips, saves or updates the current original attack sample image according to the inference result obtained by the image inference unit and the image quality score obtained by the image quality evaluation module.
5. The generation system of claim 3, wherein the partition slope represented by the mapping function is randomly initialized within the luminance range of the original image; the image mapping unit segments the luminance according to a mapping function, the luminance within each segment changing in value according to the same slope.
6. The generation system of claim 3, wherein the image inference unit is a convolutional neural network, including LeNet, ResNet, DenseNet; the regularization term in the target network module is the inverse of the average of the absolute values of the mapping function.
7. The generation system according to claim 4, wherein the image quality evaluation unit obtains an image quality score using a reciprocal of LPIPS as a scoring criterion; the higher the image quality, the lower the LPIPS value, the higher the image quality score; the image decision unit compares the image quality score of the successfully attacked image obtained by the image quality evaluation unit with the image quality score of the attack sample stored in the previous iteration, and selects the image with higher image quality score for storage; if the current image attack is unsuccessful, saving the current image as a failed image; in the image classification task, the successful attack means that the type of the attacked image is different from the type of the attacked image obtained by reasoning through the image reasoning module; the unsuccessful attack means that the type of the attacked image obtained by reasoning the image reasoning module is the same as the label.
8. An attack sample generation method implemented by the generation system according to any one of claims 1 to 7, characterized in that the generation method comprises the following steps:
the method comprises the following steps that firstly, an original image is input into an image reading module, an original image is converted into a Tensor format, an RGB color space is converted into an LAB color space, and a brightness matrix and a color matrix of the input image are obtained;
step two, an image mapping module intercepts an effective brightness matrix from the original image brightness matrix obtained in the step one, maps the effective brightness matrix by using a mapping function, and integrates the effective brightness matrix with an image invalid part after mapping is finished to obtain an original attack sample;
inputting an original attack sample into a target network module, carrying out reasoning by using an image reasoning unit and obtaining a reasoning result, and modifying a mapping function by a back propagation unit according to a classification result obtained by the reasoning and a regularization item value obtained by the regularization unit in combination with a learning rate;
step four, the image evaluation module skips, saves or updates the original attack sample obtained in the step two by selecting the inference result in the step three and the image quality score obtained by the image quality evaluation module;
and step five, repeating and iterating the steps two to four to obtain a final attack sample.
9. The generation method of claim 8, wherein in the first step, after the image reading module reads the image through an OpenCV library of python, the Numpy library is used to adjust the image format from BGR to RGB, and then the luminance information and the color information of the image are obtained through a conversion formula of an RGB space and an LAB space; converting the original picture into a Tensor format and performing through a torch library of python; the formula for converting the data in the Tensor format from the RGB color space to the LAB color space is as follows:
Figure FDA0003381276310000021
Figure FDA0003381276310000022
Figure FDA0003381276310000023
Figure FDA0003381276310000024
Figure FDA0003381276310000031
wherein, R, G and B respectively represent three channel components corresponding to the original picture in RGB color space, and the three channel components are two-dimensional matrixes with the same size as the picture; x, Y, Z are the three channel components of the representation on the XYZ color space resulting from the representation of the image on the RGB color space; xn,Yn,ZnIs a fixed value, Xn＝0.95047,Yn＝1.0,Zn1.08883; by matrix operation on the XYZ color space, a representation of the original image on the LAB color space is obtained.
10. The generation method according to claim 8, wherein in step two, the setting of the effective brightness is determined according to the maximum and minimum values of the input picture in the brightness space; in the luminance space, the part of the luminance matrix with the matrix value larger than 0 after subtracting the minimum value is the effective luminance of the input picture; after the interception of the effective brightness matrix is finished, simultaneously saving the invalid part of the image;
the mapping function is a one-dimensional vector with K parameters, i.e., K floating point numbers, as follows:
Figure FDA0003381276310000032
Figure FDA0003381276310000033
where K denotes the number of segments dividing the effective luminance, and θkDenotes the parameter, x, corresponding to the k-th segmentkBrightness value, F, representing an arbitrary position in the original imageθ(xk) And mapping the corresponding position of the image to obtain a new brightness value.
11. The generation method of claim 8, wherein in step three, the image inference unit is a convolutional neural network, and the received input is (b,3, W, H) after extracting features through a plurality of convolutional layers and a plurality of pooling layers, the input is connected with a fully-connected layer of a classification number c for classification, wherein b is a batch number, 3 represents RGB three-dimension, W is an image width, and H is an image height; in the image classification task, inputting one or b pictures which are read into a Tensor format into a target network to obtain a 1-dimensional vector representing the class of the pictures judged by the target network; the classification number of the 1-dimensional vector is c parameters which respectively represent the possibility of the class to which the picture belongs by the target network inference, wherein the class with the highest numerical value is the class of the picture obtained by the target network inference;
the network characteristics of the convolutional neural network are as follows: for the input picture, after extracting features through one or more convolution layers and pooling layers, connecting one or more full-connection layers, wherein the parameter quantity of the output layer is the target classification quantity.
12. The generation method of claim 8, wherein in step four, for an attack image generated in a certain iteration, the image inference module is put into step three to check whether the generated image can make the image inference module misclassify, and if misclassification indicates that the attack is successful; and comparing the score of the image which is successfully attacked in the image evaluation module with the corresponding score of the currently stored image, namely the attack sample stored in the previous iteration, if the score is higher, storing the new image, and if not, discarding the new image and entering the next iteration.
CN202111434878.2A 2021-11-29 2021-11-29 Image semantic anti-attack sample generation system and method based on image brightness mapping Pending CN114170443A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202111434878.2A CN114170443A (en) 2021-11-29 2021-11-29 Image semantic anti-attack sample generation system and method based on image brightness mapping

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202111434878.2A CN114170443A (en) 2021-11-29 2021-11-29 Image semantic anti-attack sample generation system and method based on image brightness mapping

Publications (1)

Publication Number Publication Date
CN114170443A true CN114170443A (en) 2022-03-11

Family

ID=80481519

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202111434878.2A Pending CN114170443A (en) 2021-11-29 2021-11-29 Image semantic anti-attack sample generation system and method based on image brightness mapping

Country Status (1)

Country Link
CN (1) CN114170443A (en)

Similar Documents

Publication Publication Date Title
CN109614996B (en) Weak visible light and infrared image fusion identification method based on generation countermeasure network
CN109859190B (en) Target area detection method based on deep learning
CN108875935B (en) Natural image target material visual characteristic mapping method based on generation countermeasure network
CN109948692B (en) Computer-generated picture detection method based on multi-color space convolutional neural network and random forest
CN111325152A (en) Deep learning-based traffic sign identification method
CN109034184B (en) Grading ring detection and identification method based on deep learning
JPH1153525A (en) Facial organ detector and medium
CN108764358A (en) A kind of Terahertz image-recognizing method, device, equipment and readable storage medium storing program for executing
CN112036260B (en) Expression recognition method and system for multi-scale sub-block aggregation in natural environment
CN112132145B (en) Image classification method and system based on model extended convolutional neural network
CN114241340A (en) Image target detection method and system based on double-path depth residual error network
CN109872326B (en) Contour detection method based on deep reinforced network jump connection
CN112580662A (en) Method and system for recognizing fish body direction based on image features
Pramunendar et al. New Workflow for Marine Fish Classification Based on Combination Features and CLAHE Enhancement Technique.
US7620246B2 (en) Method and apparatus for image processing
CN112819096A (en) Method for constructing fossil image classification model based on composite convolutional neural network
CN115937552A (en) Image matching method based on fusion of manual features and depth features
CN116563410A (en) Electrical equipment electric spark image generation method based on two-stage generation countermeasure network
CN109934835B (en) Contour detection method based on deep strengthening network adjacent connection
CN114187515A (en) Image segmentation method and image segmentation device
Jolly et al. Bringing monochrome to life: A GAN-based approach to colorizing black and white images
CN111126173A (en) High-precision face detection method
CN115359562A (en) Sign language letter spelling recognition method based on convolutional neural network
CN114170443A (en) Image semantic anti-attack sample generation system and method based on image brightness mapping
CN112560824B (en) Facial expression recognition method based on multi-feature adaptive fusion

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination