WO2024029669A1

WO2024029669A1 - Adversarial image reconstruction system and adversarial image reconstruction method

Info

Publication number: WO2024029669A1
Application number: PCT/KR2022/021137
Authority: WO
Inventors: 정민영; 정승환; 신영길
Original assignee: 숭실대학교 산학협력단; 서울대학교 산학협력단
Priority date: 2022-08-03
Filing date: 2022-12-23
Publication date: 2024-02-08
Also published as: KR102615055B1

Abstract

An adversarial image reconstruction system according to an aspect of the disclosed invention may comprise: an image reception module configured to receive an adversarial perturbed image generated by adding noise to an original image; a noise removal module configured to generate a noise-removed image by removing noise from the adversarial perturbed image by using a first artificial intelligence model; and a Fourier-based machine learning module configured to train the first artificial intelligence model via a machine learning method on the basis of a Fourier spectrum output value corresponding to the original image and a Fourier spectrum output value corresponding to the noise-removed image.

Description

Adversarial image restoration system and adversarial image restoration method

The present invention relates to a hostile image restoration system and a hostile image restoration method that can restore a hostilely modified image used in a hostile attack to a normal image.

This invention was developed by the Ministry of Science and ICT's information protection core technology development (task number: 1711152920, task number: 2021-0-00511-002, research project name: Development of Robust AI and distributed attack detection technology for edge AI security, Project management agency: Information and Communication Planning and Evaluation Institute, research period: 2022.01.01 ~ 2022.12.31) It was derived from a study conducted as part of the project. Meanwhile, there is no property interest of the Korean government in any aspect of the present invention.

An adversarial transformed image is an image created by adding adversarial perturbation that cannot be recognized by humans to the input image so that a deep neural network (DNN) for image classification misrecognizes it as a class other than the original class. It is also called an adversarial example. Methods for defending deep neural networks from such adversarial examples include the Gradient Obfuscation method, Robust Optimization method, Adversarial Example Detection method, and Gradient obfuscation method.

The Gradient Obfuscation method is a method of protecting the deep learning model from gradient-based generation methods by training the deep learning model so that it is difficult to calculate the gradient. However, these methods still have the disadvantage of being vulnerable to adversarial example generation methods.

The Robust Optimization method is a method of learning by using adversarial examples when learning a deep learning model. Robust optimization methods show good performance against adversarial attack methods, but have the problem of lowering classification accuracy for non-hostile images.

The Adversarial Example Detection method is a method of analyzing non-hostile example images and hostile example images and classifying them. There are methods for classifying input images in RGB format as non-adversarial/hostile and methods for classifying them by analyzing statistical properties such as principal component analysis, but the disadvantage is that images classified as hostile cases are discarded without being used, resulting in waste of data. There is.

The present invention provides an adversarial image restoration system and an adversarial image restoration method that can classify input images more accurately than conventional image classification methods by removing noise so that there is less difference from the original image than conventional methods when restoring adversarial images. It is for this purpose.

In addition, the present invention is intended to provide a hostile image restoration system and a hostile image restoration method that can increase the accuracy of hostile image detection by performing hostile image detection based on the image restored by removing noise from the hostile image.

Lastly, the present invention is an adversarial image restoration system and adversarial image restoration that can be used for learning an image classification deep learning model by restoring a noise-removed image based on the estimated noise without discarding the input image determined to be an adversarial image. It is intended to provide a method.

An adversarial image restoration system according to one aspect of the disclosed invention includes an image reception module configured to receive an adversarial modified image generated by adding noise to an original image; a noise removal module configured to remove noise from the adversarial deformed image using a first artificial intelligence model to generate a denoised image; And a Fourier-based machine learning module configured to learn the first artificial intelligence model through a machine learning method based on the Fourier spectrum output value corresponding to the original image and the Fourier spectrum output value corresponding to the noise-removed image. .

In addition, the Fourier-based machine learning module is an intermediate layer generated during the image classification process from an image classifier configured to classify the input image using a machine learning method that extracts features from the input image through a hierarchical structure of a plurality of stages. It may include a Fourier transform unit configured to receive the middle layer output value, which is the output value from .

In addition, the image classifier generates a first intermediate layer output value in the process of extracting features from the input original image using a machine learning method, and generates a first intermediate layer output value in the process of extracting features from the input denoised image using a machine learning method. configured to generate two middle layer output values, wherein the Fourier transform unit: performs Fourier transform on the first middle layer output value to generate a first Fourier spectrum output value representing a frequency domain; And it may be configured to perform Fourier transformation on the second middle layer output value to generate a second Fourier spectrum output value representing the frequency domain.

Additionally, the Fourier-based machine learning module may include a machine learning unit configured to learn the first artificial intelligence model through a machine learning method based on the first Fourier spectrum output value and the second Fourier spectrum output value.

In addition, the machine learning unit: calculates a loss function based on a first Fourier spectrum output value of the original image and a second Fourier spectrum output value of the noise-removed image corresponding to the original image; And the first artificial intelligence model may be configured to learn so that the loss function decreases as learning is repeated.

In addition, it may further include a hostile case classification module configured to determine whether the image to be inspected is a hostile modified image.

In addition, the image receiving module is configured to receive the inspection target image, and the noise removal module generates a noise removal inspection target image based on the inspection target image using the first artificial intelligence model learned in advance. The hostile case classification module is configured to determine whether the inspection target image is an adversarial modified image using a second artificial intelligence model, based on a composite image generated by combining the inspection target image and the noise removal inspection target image. It can be configured to determine whether or not.

In addition, the adversarial case classification module: if the inspection target image is determined to be an adversarial modified image, transmits the noise removal inspection target image to the image classifier so that the noise removal inspection target image is classified; And if the inspection target image is determined to be a normal image, the inspection target image may be transmitted to the image classifier so that the inspection target image is classified.

An adversarial image restoration method according to one aspect of the disclosed invention is a method of operating an adversarial image restoration system, comprising: receiving an adversarial modified image generated by adding noise to an original image; generating a noise-removed image by removing noise from the adversarial transformation image using a first artificial intelligence model; And a step of learning the first artificial intelligence model through a machine learning method based on the Fourier spectrum output value corresponding to the original image and the Fourier spectrum output value corresponding to the noise-removed image, and the first artificial intelligence model The learning step is: an image classifier configured to classify the input image using a machine learning method that extracts features from the input image through a hierarchical structure of a plurality of stages, and the features of the input original image are classified using a machine learning method. Generating a first intermediate layer output value in the process of extracting, and generating a second intermediate layer output value in the process of extracting features from the input noise-removed image using a machine learning method; performing Fourier transform on the first middle layer output value to generate a first Fourier spectrum output value representing a frequency domain; performing Fourier transform on the second middle layer output value to generate a second Fourier spectrum output value representing the frequency domain; And it may include learning the first artificial intelligence model through a machine learning method based on the first Fourier spectrum output value and the second Fourier spectrum output value.

Additionally, the non-transitory recording medium according to one aspect of the disclosed invention may store a computer-readable computer program to execute the hostile image restoration method.

According to one aspect of the disclosed invention, when restoring an adversarial image, noise is removed so that there is less difference from the original image than in the conventional method, so that the input image can be classified more accurately than the conventional image classification method.

Additionally, according to an embodiment of the present invention, the accuracy of hostile image detection can be improved by performing hostile image detection based on an image restored by removing noise from the hostile image.

In addition, instead of discarding the input image determined to be a hostile image, it can be restored to a noise-removed image based on the estimated noise and used for training an image classification deep learning model.

1 is a configuration diagram of an adversarial image restoration system according to an embodiment.

Figure 2 is a diagram showing which module the original image and the adversarial modified image are delivered to, according to one embodiment.

Figure 3 is a diagram illustrating a process of learning an artificial intelligence model and a process of classifying adversarial cases according to an embodiment.

FIG. 4 is a diagram illustrating the difference (FFT (Layer)) between noise (RGB) and Fourier spectrum output values corresponding to an adversarial example image according to an embodiment.

Figure 5 is a flowchart of an adversarial image restoration method according to an embodiment.

Figure 6 is a table showing the classification performance of an image classifier for noise-removed images according to an embodiment.

Figure 7 is a table showing the degree of improvement of the adversarial image restoration method according to an embodiment compared to the conventional restoration method.

FIG. 8 is a diagram illustrating classification results for a noise-removed image by an image classifier according to an embodiment.

Figure 9 is a graph showing the classification performance of a noise-removed image of an image classifier according to an embodiment.

Like reference numerals refer to like elements throughout the specification. This specification does not describe all elements of the embodiments, and general content or overlapping content between the embodiments in the technical field to which the disclosed invention pertains is omitted. The term '~unit' or '~module' used in the specification may be implemented as software or hardware, and depending on the embodiment, a plurality of '~unit' or '~module' may be implemented as a single component, or It is also possible for one '~part' or '~module' to include multiple components.

Additionally, when a part "includes" a certain component, this means that it may further include other components rather than excluding other components, unless specifically stated to the contrary. Terms such as first and second are used to distinguish one component from another component, and the components are not limited by the above-mentioned terms. Singular expressions include plural expressions unless the context clearly makes an exception.

The identification code for each step is used for convenience of explanation. The identification code does not explain the order of each step, and each step may be performed differently from the specified order unless a specific order is clearly stated in the context. there is. Hereinafter, the operating principle and embodiments of the disclosed invention will be described with reference to the attached drawings.

Referring to FIG. 1, the adversarial image restoration system 100 according to an embodiment of the present invention includes an image reception module 110, a noise removal module 120, an adversarial example classification module 130, and a Fourier-based machine learning module 140. ) and memory 150.

The adversarial image restoration system 100 may be a system configured to learn a first artificial intelligence model 151 used to remove noise from the received adversarial deformed image 400 and generate a denoised image 500. The hostile image restoration system 100 according to an embodiment of the present invention may be a system provided in a separate image restoration device or a system provided in a server.

The adversarial perturbation image 400 is an adversarial perturbation that cannot be recognized by humans on the normal original image 300 so that the deep neural network of the image classifier 200, which performs image classification, misrecognizes it as a class other than the original class. In other words, it is an image created by adding noise.

Among the techniques for analyzing or detecting images, there is a detection technique (for example, Steganalysis-based detection technique) that utilizes the characteristics of images that have similar values between neighboring pixels. Detection techniques that utilize the characteristics of images with similar values between neighboring pixels have poor detection performance for the adversarial deformed image 400 in which noise is added to the detection target image. Specifically, a detection technique that utilizes the characteristics of images with similar values between neighboring pixels determines it as a border if the difference in values in two or more directions among the eight neighboring pixels of each pixel is large. If there is noise at the border of an object in the image, This detection can be avoided.

An image modifier can generate an adversarial modified image 400 by adding noise (perturbation) to a normal image. The adversarial deformation image 400 created in this way may not recognize the boundary of the object as a boundary using a detection technique that utilizes the characteristics of images with similar values between neighboring pixels. When an adversarial modified image 400 is input to the image classifier 200, a problem may occur in which image classification performance is poor even if the image classifier 200 has good performance. In addition, the problem of intentional transfer attacks by hostile modified images 400 may occur for specific artificial intelligence models.

For any artificial intelligence model, the attacker does not know any information about the artificial intelligence model that is the target of the attack, such as network structure or learning data set, so general attacks are impossible. In other words, white-box attacks are generally impossible. However, if an attacker successfully attacks an artificial intelligence model (Model A) using a hostile modified image (400), the hostile modified image (400) will be used against another similar model (Model B) using the transferability characteristic. ), the attack can be successful. In other words, there is a problem that an adversarial example that fools one artificial intelligence model can fool other artificial intelligence models.

Therefore, before inputting the image to be classified into the image classifier 200, which uses a detection technique that utilizes the characteristics of images with similar values between neighboring pixels, if the image to be classified is an adversarial transformed image 400, the corresponding adversarial transformed image ( It is desirable to restore 400) to a normal image and input it into the image classifier 200.

The image reception module 110 may receive the adversarial modified image 400. The image reception module 110 receiving the adversarial modified images 400 may mean receiving the hostile modified images 400 input by the user through the input terminal, but is not limited to this. For example, the image reception module 110 receives the adversarial deformation images 400 previously stored in the memory 150, or the adversarial deformation image received from the server by the communication unit included in the hostile image restoration system 100 ( The hostile modified image 400 can also be received by receiving the adversarial modified image 400. The image reception module 110 may transmit the received adversarial modified image 400 to the noise removal module 120.

The adversarial modified image 400 may be an input image that is to be restored to a normal image, or may be a plurality of hostile images for learning that are input for learning the first artificial intelligence model 151 and the second artificial intelligence model 152. there is. At this time, the adversarial image for learning may be an adversarial modified image 400 generated by adding noise with previously known information to the original image 300, for which information is already known.

The noise removal module 120 may be configured to remove noise from the adversarial transformation image 400 using the first artificial intelligence model 151 to generate the noise removal image 500. The noise removal image 500 is an image generated by the adversarial image restoration system 100 based on the adversarial deformed image 400, and is used not only by the image classifier 200 of the present invention, but also by any deep image that classifies the input image using a machine learning method. It may also be transmitted to the learning model for classification of the corresponding noise-removed image 500.

The image classifier 200 may be configured to classify the input image using a machine learning method that extracts features from the input image through a hierarchical structure of a plurality of stages. For example, the image classifier 200 utilizes a CNN (Convolutional Neural Networks) structure or a DNN (Deep Neural Networks) structure that stacks several stages of convolution layers to learn a deep learning model that classifies images. It may be a classifier that does this.

Referring to FIG. 2, it can be seen through which path the data or output values of the original image 300 and the adversarial modified image 400 are transmitted to each module. For example, the adversarial modified image 400 may be passed to the noise removal module 120.

The image classifier 200 may receive the original image 300 corresponding to the adversarial modified image 400 input to the adversarial image restoration system 100. That is, the image classifier 200 can receive the original image 300 from which noise has been removed from the adversarial image for learning.

The image classifier 200 may be provided in a Fourier-based machine learning module (FFT module) 140. However, the image classifier 200 does not necessarily have to be provided in the Fourier-based machine learning module 140, and the image classifier 200 may even be provided separately outside the adversarial image restoration system 100.

Figure 3 is a diagram for explaining the process of learning an artificial intelligence model and the adversarial example classification process according to an embodiment, and Figure 4 is a diagram showing noise (RGB) and Fourier spectrum output values corresponding to a hostile example image according to an embodiment. This is a diagram showing the difference (FFT (Layer)).

Referring to FIG. 3, the Fourier-based machine learning module 140 performs a first artificial learning method using a machine learning method based on the Fourier spectrum output value corresponding to the original image 300 and the Fourier spectrum output value corresponding to the noise-removed image 500. It may be configured to learn the intelligence model 151.

The Fourier spectrum output value may be an output image obtained by Fourier transforming the middle layer output value generated in the process of classifying the input image received by the image classifier 200. That is, the Fourier spectrum output value corresponding to the original image 300 may be an output image obtained by Fourier transforming the middle layer output value generated in the process of classifying the original image 300 received by the image classifier 200. In addition, the Fourier spectrum output value corresponding to the noise-removed image 500 is the middle layer output value generated in the process of the image classifier 200 classifying the noise-removed image 500 received from the adversarial image restoration system 100. It may be an output image.

The Fourier-based machine learning module (FFT-based consistency module) 140 may include a Fourier transform unit 141 and a machine learning unit 142. The Fourier transform unit 141 may receive an intermediate layer output value, which is an intermediate layer output value generated during the image classification process, from the image classifier 200.

The image classifier 200 may generate a first intermediate layer output value in the process of extracting features from the input original image 300 using a machine learning method. For example, the first intermediate layer output value is a predetermined intermediate step generated in the process of the image classifier 200 classifying the original image 300 using a CNN structure or DNN structure that stacks several stages of convolutional layers. It may be the output value of the convolution layer.

The image classifier 200 may generate a second intermediate layer output value in the process of extracting features from the input noise-removed image 500 using a machine learning method. For example, the second intermediate layer output value is a predetermined intermediate value generated in the process of the image classifier 200 classifying the denoised image 500 using a CNN structure or DNN structure that stacks several stages of convolutional layers. It may be the convolution layer output value of the step.

The Fourier transform unit 141 may perform Fourier transform on the first middle layer output value to generate a first Fourier spectrum output value representing the frequency domain. The Fourier transform unit 141 may perform Fourier transform on the second middle layer output value to generate a second Fourier spectrum output value representing the frequency domain. That is, the Fourier transform unit 141 may perform Fast Fourier Transform (FFT) on the pixels of the received middle layer output value to generate a Fourier spectrum output value representing the frequency domain.

Referring to FIG. 4, the difference (FFT) between Fourier spectrum output values generated by the Fourier transform unit 141 based on an adversarial image (projected gradient descent (PGD)) generated by adding noise (RGB) corresponding to the adversarial example image. (Layer)) can be checked. At this time, the image represented by FFT represents the difference between two images in the Fourier image, the image represented by Layer is the difference between the first intermediate measurement output value and the second intermediate layer output value, and the image represented by FFT (Layer) represents the first Fourier spectrum output value. It may be the difference between and the second Fourier spectrum output value.

The first Fourier spectrum output value generated by the Fourier transform unit 141 may be an image representing the intensity of the frequency component constituting the first middle layer output value. Additionally, the second Fourier spectrum output value generated by the Fourier transform unit 141 may be an image representing the intensity of the frequency component constituting the second middle layer output value.

The first Fourier spectrum output value may be an image where the center of the first middle layer output value indicates a lower frequency, and the closer the center of the first middle layer output value is, the higher the frequency is. Additionally, the second Fourier spectrum output value may be an image where the center of the second middle layer output value means lower frequencies, and the closer the second middle layer output value is to the outside, the higher the frequencies are.

Operations required for the Fourier transform unit 141 to 2D transform either the first middle layer output value or the second middle layer output value to generate either the first Fourier spectrum output value or the second Fourier spectrum output value. is the same as [Equation 1].

[Equation 1]

Referring to [Equation 1], you can see how the value of each pixel of the middle layer output value is generated based on the coordinates of the corresponding pixel. At this time, M may be the horizontal size of the middle layer output value, and N may be the vertical size of the middle layer output value. The pixel value at coordinates (k,l) in the Fourier spectrum output value is based on the coordinates (k,l) in the Fourier spectrum output value and the value of each pixel (x(m,n)) in the middle layer output value [Equation 1] It can be calculated according to .

Referring again to FIGS. 1 and 3 , the noise removal module 120 uses the first artificial intelligence model 151 stored in the memory 150 to obtain information from the adversarial transformation image 400 received from the image reception module 110. Noise may be extracted and noise removed from the corresponding adversarial transformation image 400.

The noise removal module (Denoising network) 120 extracts and removes noise using a first artificial intelligence model 151 previously learned based on feature data extracted from the adversarial deformed image 400. This may be extracting noise and removing noise from the adversarial transformation image 400. At this time, a CNN (Convolutional Neural Networks) structure that stacks several stages of convolutional layers can be used to learn how to extract features from the adversarial deformed image 400, and in particular, U-net can be used. However, the method of extracting features from the adversarial modified image 400 is not limited to this. The characteristics of a specific adversarial deformed image 400 may be information representing various characteristics of the adversarial deformed image 400. For example, the characteristics of a specific adversarial deformation image 400 may be information about color, brightness, border, etc. in each pixel unit of the adversarial deformation image 400, but are not limited thereto.

In order for the noise removal module 120 to remove noise using the first artificial intelligence model 151, the first artificial intelligence model 151 needs to be trained in advance based on a plurality of hostile images for training. The machine learning unit 142 may be configured to learn the first artificial intelligence model 151 through a machine learning method based on the first Fourier spectrum output value and the second Fourier spectrum output value.

Machine learning can mean using a model composed of multiple parameters and optimizing the parameters with given data. Machine learning may include supervised learning, unsupervised learning, and reinforcement learning, depending on the type of learning problem. Supervised learning is learning the mapping between input and output, and can be applied when input and output pairs are given as data. Unsupervised learning is applied when there is only input and no output, and can find regularities between inputs. However, machine learning according to one embodiment is not necessarily limited to the above-described learning method.

Specifically, the machine learning unit 142 performs a loss function (f) based on the first Fourier spectrum output value of the original image 300 and the second Fourier spectrum output value of the noise-removed image 500 corresponding to the original image 300. denoising) can be calculated. The machine learning unit 142 may learn the first artificial intelligence model 151 so that the loss function (f denoising) decreases as learning is repeated. The machine learning unit 142 can learn the first artificial intelligence model 151 through repetitive machine learning. The first artificial intelligence model 151 may be a deep learning model. The first artificial intelligence model 151 to be learned may be stored in the memory 150.

The loss function (f denoising) may be a loss function (MSE Loss) used in regression analysis of the mean squared error (MSE) for the first Fourier spectrum output value and the second Fourier spectrum output value. The machine learning unit 142 may be configured to learn the first artificial intelligence model 151 so that the loss function (f denoising) decreases as learning is repeated. In other words, the more you repeat learning, the closer the second Fourier spectrum output value output by the machine learning unit 142 is to the first Fourier spectrum output value, which means that the more you repeat learning, the more noise generated through the first artificial intelligence model 151 becomes. This may mean that the removed image 500 is closer to the original image 300.

In this way, the adversarial image restoration system 100 learns the first artificial intelligence model 151 in advance, and uses the first artificial intelligence model 151 to restore the adversarial deformed image 400 to the normal denoised image 500. can do. The noise-removed image 500 restored in this way can be transmitted to the image classifier 200 for classification instead of the adversarial transformation image 400 before restoration.

However, in the case of a denoising image (500) generated by inputting the inspection target image into the adversarial image restoration system (100) in a situation where it is not known whether the inspection target image is an adversarial modified image (400) or a normal original image (300), It may not be desirable to transmit it as is to the image classifier 200. If the image to be inspected is a normal original image 300, the noise-removed image 500 of the original image 300 may be somewhat different from the original image 300. In this situation, it is preferable that the normal original image 300 is directly input to the image classifier 200 instead of the noise-removed image 500 of the original image 300. Therefore, it is necessary to determine whether the image to be inspected to be input to the image classifier 200 is a hostile modified image 400 or a normal image, and to determine the image to be transmitted to the image classifier 200 according to the judgment result.

The image receiving module 110 may receive an image to be inspected. The image receiving module 110 may transmit the image to be inspected to the noise removal module 120. The noise removal module 120 may generate a noise removal inspection target image based on the inspection target image using the first artificial intelligence model 151 learned in advance. The noise removal module 120 may transmit the inspection target image and the noise removal inspection target image to the adversarial case classification module 130.

The adversarial case classification module 130 may generate a composite image by combining the inspection target image and the noise removal inspection target image. The composite image may be an image that includes all three RGB image information for the image to be inspected and all three RGB image information for the image to be inspected for noise removal.

The hostile case classification module 130 may be configured to determine whether the image to be inspected is a hostile modified image 400. Specifically, the hostile case classification module 130 may determine whether the image to be inspected is a hostile modified image 400 using the second artificial intelligence model 152 based on the generated synthetic image.

The adversarial case classification module 130 determines whether the image to be inspected is a hostile modified image by using the second artificial intelligence model 152 learned in advance based on feature data extracted from the synthetic image. . At this time, a CNN structure that stacks several stages of convolutional layers can be used to learn how to extract features from synthetic images, and in particular, logistic regression can be used, but it is not possible to determine whether the image to be inspected is an adversarial transformation image. The method of judging is not limited to this.

The machine learning unit 142 generates a loss function (f cls) used to determine whether or not it is a hostile image based on feature data extracted from a plurality of inspection target images and information on whether each inspection target image is actually an adversarial transformed image. It can be calculated. The loss function (f cls) used to determine whether an image is hostile is a loss function used for binary classification of hostile noise and may be a cross-entropy loss function. The machine learning unit 142 may be configured to learn the second artificial intelligence model 152 so that the loss function (f cls) used to determine whether a hostile image is decreased as learning is repeated.

If the hostile case classification module 130 determines that the image to be inspected is a hostile modified image 400, the image to be inspected for noise removal may be transmitted to the image classifier 200 so that the image to be inspected for noise removal is classified. At this time, the image classifier 200 may classify the image subject to noise removal inspection.

If the hostile case classification module 130 determines that the image to be inspected is a normal image, it may transmit the image to be inspected to the image classifier 200 so that the image to be inspected is classified. At this time, the image classifier 200 may classify the image to be inspected.

At least one component may be added or deleted in response to the performance of the components described above. Additionally, it will be easily understood by those skilled in the art that the mutual positions of the components may be changed in response to the performance or structure of the system.

Figure 5 is a flowchart of an adversarial image restoration method according to an embodiment. This is only a preferred embodiment for achieving the purpose of the present invention, and of course, some components may be added or deleted as needed.

Referring to FIG. 5, the image reception module 110 may receive an adversarial modified image 400 (1001). The image reception module 110 may transmit the adversarial modified image 400 to the noise removal module 120.

The noise removal module 120 may remove noise from the adversarial transformation image 400 using the first artificial intelligence model 151 to generate the noise removal image 500 (1002). The noise removal module 120 may transmit the generated noise removal image 500 to the image classifier 200.

The image classifier 200 generates a first intermediate layer output value in the process of extracting features from the input original image 300 using a machine learning method, and extracts features from the input denoised image 500 using a machine learning method. In the process, a second middle layer output value can be generated (1003). The image classifier 200 may transmit the first middle layer output value and the second middle layer output value to the Fourier transform unit 141. That is, the Fourier transform unit 141 can receive the middle layer output value, which is the output value of the middle layer generated during the image classification process.

The Fourier transform unit 141 performs Fourier transform on the first middle layer output value to generate a first Fourier spectrum output value representing the frequency domain, and Fourier transforms the second middle layer output value to generate a second Fourier spectrum output value representing the frequency domain. Can (1004). The Fourier transform unit 141 may transmit the first Fourier spectrum output value and the second Fourier spectrum output value to the machine learning unit 142.

The machine learning unit 142 may learn the first artificial intelligence model 151 through a machine learning method based on the first Fourier spectrum output value and the second Fourier spectrum output value (1005).

The image receiving module 110 may receive an image to be inspected (1006). The image receiving module 110 may transmit the image to be inspected to the noise removal module 120.

The noise removal module 120 may generate a noise removal inspection target image based on the inspection target image using the pre-trained first artificial intelligence model 151 (1007). The noise removal module 120 may transmit the image subject to noise removal inspection to the adversarial case classification module 130.

The hostile case classification module 130 uses the second artificial intelligence model 152 to determine whether the image to be inspected is an adversarial modified image 400 based on a composite image created by combining the image to be inspected and the image to be inspected for noise removal. You can determine whether or not (1008).

The adversarial example classification module 130 may transmit an image on which image classification is to be performed to the image classifier 200 (1009). At this time, if the hostile case classification module 130 determines that the image to be inspected is a hostile modified image 400, it may transmit the image to be inspected for noise removal to the image classifier 200 so that the image to be inspected for noise removal is classified. Additionally, if the hostile case classification module 130 determines that the image to be inspected is a normal image, it may transmit the image to be inspected to the image classifier 200 so that the image to be inspected is classified.

The image reception module 110, the noise removal module 120, the adversarial example classification module 130, the Fourier-based machine learning module 140, the Fourier transform unit 141, and the machine learning unit 142 are an adversarial image restoration system ( 100) may include any one processor among a plurality of processors included in the processor. Additionally, the adversarial image restoration method according to the embodiment of the present invention described so far may be implemented in the form of a program that can be driven by a processor.

Here, the program may include program instructions, data files, and data structures, etc., singly or in combination. Programs may be designed and produced using machine code or high-level language code. The program may be specially designed to implement the above-described method for modifying the code, or may be implemented using various functions or definitions known and available to those skilled in the art in the field of computer software. A program for implementing the above-described information display method may be recorded on a non-transitory recording medium readable by a processor. At this time, the recording medium may be the memory 150.

The memory 150 can store programs that perform the operations described above and the operations described later, and the memory 150 can execute the stored programs. In the case where there are a plurality of processors and memories 150, they may be integrated into one chip or may be provided in physically separate locations. The memory 150 may include volatile memory such as Static Random Access Memory (S-RAM) or Dynamic Random Access Memory (D-Lab) for temporarily storing data. In addition, the memory 150 includes read only memory (ROM), erasable programmable read only memory (EPROM), and electrically erasable programmable read only memory (EEPROM) for long-term storage of control programs and control data. May include non-volatile memory. The processor may include various logic circuits and operation circuits, process data according to a program provided from the memory 150, and generate control signals according to the processing results.

In order to verify the performance of the adversarial image restoration system 100 according to an embodiment of the present invention, images of the CIFAR-10 and CIFAR-100 datasets are used by the conventional adversarial image classification method and the adversarial image restoration method of the present invention. An experiment was conducted to classify and restore .

FIG. 6 is a table showing the classification performance of the image classifier for the noise removal image 500 according to an embodiment, and FIG. 7 shows the degree to which the adversarial image restoration method according to an embodiment is improved compared to the conventional restoration method. It is a table, and FIG. 8 is a diagram showing classification results for a noise-removed image 500 of an image classifier according to an embodiment, and FIG. 9 is a diagram showing classification of a noise-removed image 500 of an image classifier according to an embodiment. This is a graph showing performance.

Referring to FIG. 6, the classification results for the restored image can be confirmed according to one embodiment. In order to verify the performance of the restored image according to one embodiment, a PGD (Projected gradient descent) attack method that generates adversarial examples by iteratively calculating the gradient for the loss value of the adversarial deformed image 400 and the boundary between classes is used. A Deepfool attack method that finds and generates noise in the direction of the nearest boundary line from the adversarial deformed image (400), a three-generation method using three methods to measure the size of the adversarial noise, and a highly reliable adversarial example using an effective objective function. Image restoration was performed on the adversarial modified images 400 generated by the C&W (Carlini & Wagner) attack method using a restoration method according to one embodiment and another restoration method. Referring to the table, the adversarial image classification method (FFT (Layer (x ori & x pred))) according to one embodiment is different from other methods (x ori & x pred, FFT (x ori & It can be seen that the classification performance is better than ori & x pred)). Specifically, according to one embodiment, the image restored by an adversarial image restoration method (FFT (Layer(x ori & & x pred), which has better classification performance than the image restored based on Fourier transform (FFT(x ori & Confirmed.

Referring to FIG. 7, according to one embodiment, the performance when classifying hostile images and non-hostile images based on features extracted from the restored image and the input image can be confirmed. At this time, feature values were extracted from the input image using four methods (LID, Mahalanobis, LayerMFS, LayerPFS). In particular, the method of (LID w/ ours, Mahalanobis w/ ours, LayerMFS w/ ours, LayerPFS w/ ours) extracts feature values from a composite image that combines the input image and the restored denoised image (500). As a result of performing adversarial/non-adversarial binary classification based on the feature values extracted in this way, it can be seen that the performance of the adversarial image restoration method according to one embodiment is superior.

Referring to Figure 8, you can see the results of classifying the input image by the target classifier of the hostile image generation method. At this time, “Adversarial” indicates the result when the adversarial deformed image (400) is classified using a deep learning model, and “Denoised” indicates the result when only the denoised images (500) restored using an adversarial denoising network are classified. . At this time, it can be confirmed that the classification result for the “Denoised” image is the same as the classification result for the “Original” image to which no hostile transformation has been applied. That is, it can be confirmed that the attack target model normally performs image classification without any problem with respect to the image restored by the adversarial modified image restoration method according to one embodiment.

Referring to FIG. 9, the adversarial modified image 400 of the CIFAR-10 and CIFAR-100 datasets is restored using the conventional adversarial image detection method and the adversarial image restoration method of the present invention to create an attack target model (target classifier). The classified results can be confirmed with a confusion matrix. At this time, “Adversarial” represents the result when the adversarial deformed image 400 is classified using a deep learning model, and “Denoised” refers to the result when only images restored through the adversarial deformed image restoration method according to one embodiment are classified. It is shown. Specifically, the results of an experiment in which the image classification deep learning model, which is the target of the attack, classifies images of the CIFAR-10 and CIFAR-100 datasets modified by the PGD (Projected gradient descent) attack method and the Deepfool attack method can be seen in a graph. At this time, the more clearly the diagonal component appears, the more likely it is that the image classification deep learning model that is the target of the attack has properly classified the image. In other words, it can be confirmed that the image classification deep learning model that is the target of the attack did not perform image classification well for the adversarial modified images (Adversarial), but the noise removal image generated through the adversarial image restoration method according to one embodiment ( Denoised (500), you can see the diagonal trend in the graph. That is, even if the original image 300 is hostilely modified due to a hostile attack, according to one embodiment, the denoised image 500 generated based on the hostile modified image 400 is an image classification deep learning model that is the target of the attack. It can be confirmed that it can be used without problem to perform image classification.

As described above, the disclosed embodiments have been described with reference to the attached drawings. A person skilled in the art to which the present invention pertains will understand that the present invention can be practiced in forms different from the disclosed embodiments without changing the technical idea or essential features of the present invention. The disclosed embodiments are illustrative and should not be construed as limiting.

Claims

an image receiving module configured to receive an adversarial transformed image generated by adding noise to the original image;

a noise removal module configured to remove noise from the adversarial deformed image using a first artificial intelligence model to generate a denoised image; and

Adversarial image restoration comprising a Fourier-based machine learning module configured to learn the first artificial intelligence model through a machine learning method based on the Fourier spectrum output value corresponding to the original image and the Fourier spectrum output value corresponding to the noise-removed image. system.
According to paragraph 1,

The Fourier-based machine learning module is,

To receive the middle layer output value, which is the output value from the middle layer generated during the image classification process, from an image classifier configured to classify the input image using a machine learning method that extracts features from the input image through a hierarchical structure of multiple stages. An adversarial image restoration system comprising a Fourier transform.
According to paragraph 2,

The image classifier generates a first intermediate layer output value in the process of extracting features from the input original image using a machine learning method, and generates a second intermediate layer output value in the process of extracting features from the input denoised image using a machine learning method. configured to generate a layer output,

The Fourier transform unit:

Performing Fourier transform on the first middle layer output value to generate a first Fourier spectrum output value representing a frequency domain; and

An adversarial image restoration system configured to Fourier transform the second middle layer output to generate a second Fourier spectral output representing a frequency domain.
According to paragraph 3,

The Fourier-based machine learning module is,

An adversarial image restoration system comprising a machine learning unit configured to learn the first artificial intelligence model through a machine learning method based on the first Fourier spectrum output value and the second Fourier spectrum output value.
According to paragraph 4,

The machine learning department:

calculating a loss function based on a first Fourier spectrum output value of the original image and a second Fourier spectrum output value of the noise-removed image corresponding to the original image; and

An adversarial image restoration system configured to learn the first artificial intelligence model so that the loss function decreases as learning is repeated.
According to clause 5,

An adversarial image restoration system, further comprising an adversarial case classification module configured to determine whether the image to be inspected is an adversarial modified image.
According to clause 6,

The image receiving module is,

Configured to receive the image to be inspected,

The noise removal module,

Configured to generate a noise-removing inspection target image based on the inspection target image using the first artificial intelligence model learned in advance,

The hostile case classification module is,

An adversarial image restoration system configured to determine whether the inspection target image is an adversarial modified image using a second artificial intelligence model, based on a composite image generated by combining the inspection target image and the noise removal inspection target image. .
In clause 7,

The adversarial case classification module:

If the inspection target image is determined to be an adversarial modified image, transmitting the noise removal inspection target image to the image classifier so that the noise removal inspection target image is classified; and

When the inspection target image is determined to be a normal image, an adversarial image restoration system transmits the inspection target image to the image classifier so that the inspection target image is classified.
As a method of operating an adversarial image restoration system,

Receiving an adversarial transformed image generated by adding noise to the original image;

generating a noise-removed image by removing noise from the adversarial transformation image using a first artificial intelligence model; and

Comprising learning the first artificial intelligence model through a machine learning method based on the Fourier spectrum output value corresponding to the original image and the Fourier spectrum output value corresponding to the noise-removed image,

The steps for learning the first artificial intelligence model are:

In the process of extracting features from the input original image using a machine learning method, an image classifier is configured to classify the input image using a machine learning method that extracts features from the input image through a plurality of levels of hierarchy. Generating a first intermediate layer output value and generating a second intermediate layer output value in the process of extracting features from the input noise-removed image using a machine learning method;

performing Fourier transform on the first middle layer output value to generate a first Fourier spectrum output value representing a frequency domain;

performing Fourier transform on the second middle layer output value to generate a second Fourier spectrum output value representing the frequency domain; and

An adversarial image restoration method comprising learning the first artificial intelligence model through a machine learning method based on the first Fourier spectrum output value and the second Fourier spectrum output value.
A non-transitory recording medium storing a computer-readable computer program to execute the hostile image restoration method of claim 9.