KR102663350B1 - Adversarial perturbation image detection method and computer program stored in a recording medium to execute the method - Google Patents

Abstract

An adversarial modified image detection method according to an embodiment of the disclosed invention includes the steps of receiving an image to be inspected by an input unit; Extracting, by a pixel color information extraction unit, a color information value that is an integer value of one of unique integers that are integers within a bit depth value range of the image to be inspected for each pixel of the image to be inspected; and calculating, by an information entropy calculation unit, a color information entropy value of the image to be inspected based on color information values for each pixel of the image to be inspected.

Description

A computer program stored on a recording medium to execute a hostile modified image detection method and a hostile modified image detection method {ADVERSARIAL PERTURBATION IMAGE DETECTION METHOD AND COMPUTER PROGRAM STORED IN A RECORDING MEDIUM TO EXECUTE THE METHOD}

The present invention relates to an adversarial modified image detection method that can determine whether an image used in image analysis or machine learning is a hostilely modified image.

An adversarial transformed image refers to an image created by adding adversarial perturbation that cannot be recognized by humans to the input image so that a deep neural network for image classification misrecognizes it as a class other than the original class. , It is also called an adversarial example.

The Feature Squeezing technique, a representative adversarial example detection method, is the difference between the output value of the DNN model for the input image and the output value of the DNN model for the input image to which bit depth reduction, median smoothing, and non-local smoothing were applied, respectively. Detect adversarial examples using .

However, methods such as feature squeezing that use the difference in the output value of the DNN model by modulating the input image are a method of mitigating, not removing, the adversarial deformation included in the adversarial example, rather than removing the adversarial deformation included in the heavily modulated adversarial example. In the case of the example, even if the adversarial transformation is alleviated by reducing the bit depth, median smoothing, and non-local smoothing, the DNN model is affected by the adversarial transformation and the difference in the output value of the DNN model before and after modulation is not large, making it difficult to detect adversarial examples well. there is a problem.

Additionally, in the inference stage, the defender cannot know whether the data input to the DNN model is normal or an attack, so all input data must be checked to see whether it is normal or an attack. Therefore, the method using the output result of the DNN model does not detect adversarial attacks well. Not only is it not possible, but there is a problem that it takes a long time to detect attack data.

The purpose of the present invention is to provide an adversarial modified image detection method and computer program that can detect adversarial examples without using the output value of a DNN model and thus has superior performance in detecting adversarial attacks.

In addition, the present invention is to provide a method and computer program for detecting adversarial modified images that can automatically classify an image to be inspected as a normal image or an adversarial attack image at a faster rate than a method using the output value of a DNN model. .

In addition, the present invention is intended to provide an adversarial modified image detection method and computer program that can be used to detect deepfake images generated by a GAN (Generative Adversarial Network) as well as detecting adversarial examples generated by various adversarial attacks. .

An adversarial modified image detection method according to one aspect of the disclosed invention includes the steps of receiving an image to be inspected by an input unit; Extracting, by a pixel color information extraction unit, a color information value that is one integer value among values within a bit depth value range of the image to be inspected for each pixel of the image to be inspected; and calculating, by an information entropy calculation unit, a color information entropy value of the image to be inspected based on color information values for each pixel of the image to be inspected.

In addition, comparing the color information entropy value of the inspection target image with a reference entropy value by an adversarial deformed image determination unit; and classifying, by the hostile modified image determination unit, the image to be inspected as an adversarially modified image if the color information entropy value of the image to be inspected is greater than or equal to the reference entropy value.

In addition, the step of calculating the color information entropy value of the image to be inspected includes calculating, by the information entropy calculation unit, the number of pixels of the same color, which is the number of pixels having the same unique integer as a color information value, in the image to be inspected. Deriving each unique integer and dividing the number of same-color pixels of each unique integer by the number of pixels of the image to be inspected to calculate a ratio of same-color pixels falling within a range of 0 to 1 of each unique integer; and calculating, by the information entropy calculation unit, a color information entropy value of the image to be inspected based on logarithmic function application values derived by taking a logarithmic function to the ratio of pixels of the same color of each unique integer of the image to be inspected. may include.

Additionally, generating a modified image by reducing the bit depth value of the image to be inspected to a predetermined bit value by a modified image generator; and calculating, by the information entropy calculation unit, a color information entropy value of the modified image based on the color information values for each pixel of the modified image.

Additionally, calculating a difference between a color information entropy value of the image to be inspected and a color information entropy value of the modified image, by an adversarial modified image determination unit; and classifying the inspection target image as a hostile modified image, by the hostile modified image determination unit, if the difference between the color information entropy value of the inspection target image and the color information entropy value of the modified image is greater than or equal to a reference difference value. It can be included.

In addition, the step of calculating the color information entropy value of the modified image includes calculating the number of pixels of the same color, which is the number of pixels having the same unique integer as a color information value in the modified image, by the information entropy calculation unit, and dividing the number of pixels of the same color into each unique integer. calculating a ratio of pixels of the same color falling within a range of 0 to 1 of each unique integer by dividing the number of pixels of the same color in each unique integer by the number of pixels in the deformed image; And calculating, by the information entropy calculation unit, a color information entropy value of the transformed image based on logarithmic function application values derived by taking a logarithmic function for the same color pixel ratio of each unique integer of the transformed image. can do.

Additionally, receiving a plurality of normal images as input through the input unit; calculating, by the information entropy calculation unit, a color information entropy value of the normal image based on color information values for each pixel of the normal image; generating a modified normal image by reducing the bit depth value of the normal image to a predetermined bit value, by the modified image generator; calculating a color information entropy value of the deformed normal image based on the color information values for each pixel of the deformed normal image, by the information entropy calculation unit; calculating, by a reference difference value determination unit, a difference between a color information entropy value of the normal image and a color information entropy value of the modified normal image; and the reference difference value determination unit, when the difference values of the color information entropy values of the plurality of normal images are arranged in order of size, the difference in the color information entropy value of the normal image corresponding to the pre-designated upper order is calculated as the reference difference. A step of determining a value may be further included.

Additionally, generating the modified image may include generating the modified image by reducing the bit depth value of the image to be inspected by 1, by the modified image generator.

In addition, the step of storing the image classified as the hostile modified image separately from the remaining images to be inspected by the hostile modified image determination unit so that the hostile modified image is not input to the image analysis device may be further included. .

A computer program according to one aspect of the disclosed invention may be stored in a computer-readable recording medium to execute the adversarial modified image detection method.

According to one aspect of the disclosed invention, adversarial examples can be detected without utilizing the output value of the DNN model, so detection performance of whether or not a hostile attack is present can be better.

Additionally, according to an embodiment of the present invention, an image to be inspected can be automatically classified as a normal image or a hostile attack image at a faster rate than a method using the output value of a DNN model.

Additionally, according to an embodiment of the present invention, it can be used not only to detect adversarial examples generated by various adversarial attacks, but also to detect DeepFake images generated by a Generative Adversarial Network (GAN).

1 is a control block diagram of an adversarial deformed image detection system according to an embodiment.
Figure 2 is a diagram illustrating the creation process of an adversarial deformed image.
Figure 3 is a diagram for explaining a modified image according to an embodiment.
Figure 4 is a diagram to explain the distribution of pixel ratios of the same color between a normal image and an adversarially modified image.
Figure 5 is another diagram to explain the distribution of pixel ratios of the same color between a normal image and an adversarially modified image.
FIG. 6 is a diagram illustrating a process of separately classifying an inspection target image determined to be an adversarial modified image according to an embodiment.
Figure 7 is a diagram for explaining a reference difference value according to an embodiment.
Figure 8 is a flowchart of an adversarial modified image detection method according to one embodiment.
Figure 9 is a graph showing the average color information entropy of a normal image and an adversarial modified image according to an embodiment.
Figure 10 is a graph showing classification accuracy according to the bit depth value of the modified image.

Like reference numerals refer to like elements throughout the specification. This specification does not describe all elements of the embodiments, and general content or overlapping content between the embodiments in the technical field to which the disclosed invention pertains is omitted. The term '~unit' used in the specification may be implemented as software or hardware, and depending on the embodiments, multiple '~units' may be implemented as one component, or one '~unit' may be implemented as a plurality of components. It is also possible to include elements.

Additionally, when a part "includes" a certain component, this means that it may further include other components rather than excluding other components, unless specifically stated to the contrary.

Terms such as first and second are used to distinguish one component from another component, and the components are not limited by the above-mentioned terms.

Singular expressions include plural expressions unless the context clearly makes an exception.

The identification code for each step is used for convenience of explanation. The identification code does not explain the order of each step, and each step may be performed differently from the specified order unless a specific order is clearly stated in the context. there is.

Hereinafter, the operating principle and embodiments of the disclosed invention will be described with reference to the attached drawings.

FIG. 1 is a control block diagram of an adversarial deformed image detection system according to an embodiment, and FIG. 2 is a diagram illustrating a process of generating an adversarial deformed image.

Referring to FIG. 1, the hostile deformed image detection system 100 determines whether the input inspection target image 200 is a normal image 500 or an adversarial modified image 400 and classifies the inspection target image 200. there is.

The inspection target image 200 may be an image for which it is determined whether or not it contains a hostile attack before being input to the image analysis device 600 in the form of image data.

The image analysis device 600 may be a device that recognizes or analyzes objects included in an input image, but is not limited thereto, and may be a device that uses the input image in other ways. For example, the image analysis device 600 may use the input image to learn an artificial intelligence model through machine learning. Machine learning can mean using a model composed of multiple parameters and optimizing the parameters with given data. Conversely, the image analysis device 600 may be a device that analyzes an input image using an already learned artificial intelligence model.

The hostile modified image detection system 100 according to an embodiment of the present invention may be a system provided in a separate hostile modified image detection device or may be a system provided in a server.

Referring to FIG. 2, an example of the generation process of the adversarial deformed image 400 can be seen.

Among the techniques for analyzing or detecting images, there is a detection technique (for example, Steganalysis-based detection technique) that utilizes the characteristics of images that have similar values between neighboring pixels.

Detection techniques that utilize the characteristics of images with similar values between neighboring pixels will have poor detection performance if noise is added to the image to be detected.

Specifically, a detection technique that utilizes the characteristics of images with similar values between neighboring pixels determines it as a border if the difference in values in two or more directions among the eight neighboring pixels of each pixel is large. If there is noise at the border of an object in the image, This detection can be avoided.

An adversarial deformed image 400 may be created by adding noise (perturbation) to the normal image 500.

The adversarial deformation image 400 created in this way may not recognize the boundary of the object as a boundary using a detection technique that utilizes the characteristics of images with similar values between neighboring pixels. When an adversarial modified image 400 is input to the image analysis device 600, detection performance deteriorates even if the image analysis device 600 has good performance, and if the image is used for machine learning, the performance of the generated artificial intelligence model decreases. Falling problems may occur. Therefore, before the image to be analyzed is input to the image analysis device 600, it is necessary to detect whether the image has been hostilely modified in advance.

Referring again to FIG. 1, the adversarial deformed image detection system 100 includes an input unit 110, a pixel color information extraction unit 120, an information entropy calculation unit 130, an adversarial deformed image determination unit 140, and a deformed image generation unit. It may include a unit 150 and a reference difference value determination unit 160.

The input unit 110 may receive an image 200 to be inspected. The method in which the input unit 110 receives the inspection target image 200 may be that a communication module provided in the input unit 110 receives the inspection target image 200 from another component outside the system, but the input unit 110 receives the inspection target image 200. The method of receiving the target image 200 is not limited to this.

The adversarial deformed image detection system 100 may determine whether the inspected image 200 is an adversarial deformed image 400 with noise added based on the color information of each pixel of the input inspected image 200.

The bit depth value of an image may be the number of color information that can be expressed per pixel. For example, if the bit depth value of an image is 8 bits, the colors that can be expressed by one pixel can be 256 each for red, green, and blue. For example, if the color of one pixel is blue, that pixel can express one blue color out of a total of 256 blue colors, from the brightest blue to the darkest blue.

That is, if the bit depth value of an image is 8 bits, each pixel of the image can have one color information out of 256 color information for all colors of red, green, and blue. For example, the color information (R, G, B) value of one pixel may be (20, 254, 132).

The pixel color information extraction unit 120 may extract a color information value for each pixel of the image 200 to be inspected.

The color information value may be an integer value corresponding to each pixel of the image 200 to be inspected.

Specifically, the color information value may be an integer value among unique integers.

The unique integer may be an integer within the bit depth value range of the image 200 to be inspected.

For example, if the bit depth value of the image 200 to be inspected is 8 bits, the unique integer of the image may be a total of 256 integer values between 0 and 255. At this time, the color information value of each pixel may be an integer value between 0 and 255.

Information entropy or Shannon entropy is the average value of information contained in a message. Specifically, information entropy can be defined as the negative logarithm of the probability distribution of possible events or messages.

The information entropy calculation unit 130 may derive the number of pixels of the same color for each unique integer.

The number of pixels of the same color may be the number of pixels that have the same unique integer as a color information value in the image 200 to be inspected. For example, if the number of pixels having 255 as a color information value in an inspection target image 200 with a bit depth value of 8 bits is 4, the number of pixels of the same color for 255, which is a unique integer, may be 4.

The information entropy calculator 130 may divide the number of pixels of the same color in each unique integer by the number of pixels in the image to be inspected 200 to calculate the ratio of pixels of the same color within the range of 0 to 1 for each unique integer. For example, in an image to be inspected (200) with an image size of 224 width, 224 height, and 3 color channels (224x224x3) and a bit depth value of 8 bits, the number of pixels of the same color for a unique integer of 255 is 4. , the ratio of same color pixels to the unique integer of 255 may be 4/(224x224x3).

The information entropy calculation unit 130 calculates the color information entropy value of the image to be inspected 200 based on the logarithmic function application values derived by taking a logarithmic function to the ratio of pixels of the same color of each unique integer of the image to be inspected 200. It can be calculated.

[Equation 1]

Specifically, referring to [Equation 1], sum(pk) is the number of pixels of the image to be inspected (200), pki is the ratio of pixels of the same color to the unique integer i, and S is the color information of the image to be inspected (200). It could be entropy.

The information entropy calculator 130 may determine the absolute value of the sum of all logarithmic function application values for each derived unique integer as the color information entropy of the image 200 to be inspected.

For example, if the number of pixels of the same color for the unique integers 0, 1, 2, and 255 is 3, 5, 1, and 4, respectively, in an image to be inspected (200) where the number of pixels is 10000, then the unique integers 0, 1, and 2 and 255, the same color pixel ratios may be 0.0003, 0.0005, 0.0001, and 0.0004. The absolute value of the sum of all logarithmic function application values derived by taking a logarithmic function to each of the pixel ratios of the same color may be the color information entropy of the image 200 to be inspected.

The adversarial modified image 400 is created by adding noise to a normal image. At this time, each pixel of the adversarial modified image 400 has color information that is changed differently from the color information value of each pixel of the original normal image.

In particular, the adversarial modified image 400 has relatively less continuity in the distribution of color information values for each pixel than a normal image. That is, in the adversarial modified image 400, the number of pixels with specific color information values may be very small compared to pixels with different color information values.

In summary, in the adversarial deformed image 400, the number of same-color pixels of specific unique integers may be much smaller than the number of same-color pixels of other unique integers, and the ratio of identical color pixels of the unique integers may also be the ratio of identical color pixels of other unique integers. It can be closer to 0. Since the logarithmic function application values increase as the positive number approaches 0, the adversarial transformation image 400 may have a larger color information entropy value than the normal image 500.

The adversarial transformation image determination unit 140 may compare the color information entropy value of the inspection target image 200 with the reference entropy value. The reference entropy value may be a color information entropy value that serves as a standard for determining whether the color information entropy value of the inspection target image 200 corresponds to the color information entropy value of the hostile transformation image 400.

The hostile modified image determination unit 140 may classify the inspected image 200 as a hostile modified image 400 if the color information entropy value of the inspected image 200 is greater than or equal to the reference entropy value.

The input unit 110, the pixel color information extraction unit 120, the information entropy calculation unit 130, the adversarial transformation image determination unit 140, the transformation image generation unit 150, and the reference difference value determination unit 160 are used for adversarial transformation. It may include any one processor among a plurality of processors included in the image detection system 100. Additionally, the method for detecting an adversarial modified image according to the embodiments of the present invention described so far and the embodiments to be described in the future may be implemented in the form of a program that can be driven by a processor.

Here, the program may include program instructions, data files, and data structures, etc., singly or in combination. Programs may be designed and produced using machine code or high-level language code. The program may be specially designed to implement the above-described method for modifying the code, or may be implemented using various functions or definitions known and available to those skilled in the art in the computer software field. A program for implementing the above-described information display method may be recorded on a recording medium readable by a processor. At this time, the recording medium may be memory.

The memory can store programs that perform the operations described above and the operations described later, and the memory can execute the stored programs. In the case where there are multiple processors and memories, it is possible for them to be integrated into one chip or to be provided in physically separate locations. The memory may include volatile memory such as SRAM (Static Random Access Memory, S-RAM) and D-Lab (Dynamic Random Access Memory) for temporarily storing data. In addition, memory is non-volatile memory such as Read Only Memory (ROM), Erasable Programmable Read Only Memory (EPROM), and Electrically Erasable Programmable Read Only Memory (EEPROM) for long-term storage of control programs and control data. may include.

The processor may include various logic circuits and operation circuits, process data according to a program provided from memory, and generate control signals according to the processing results.

Although it is common for the color information entropy value of the hostile modified image (400) to be larger than that of the normal image (500), the color information entropy value of the hostile modified image (400) is necessarily higher than that of the normal image (500). This isn't that big of a deal. For example, the color information entropy value of the normal image 500 may be relatively larger than the color information entropy values of other normal images 500 due to the color information values of the pixels of the normal image 500. Considering this case, there is a problem that the adversarial deformed image 400 cannot be completely classified simply by the size of the color information entropy value. Therefore, a method for detecting the adversarial deformed image 400 is needed regardless of the size of the color information entropy of the input adversarial image.

Figure 3 is a diagram for explaining a modified image according to an embodiment.

Referring to FIG. 3, a modified image 300 can be generated by changing the bit depth value of the image 200 to be inspected. Here, the modified image 300 may be an image expressed by varying the bit depth value of the original image, completely unrelated to the hostile modified image 400.

It can be seen that as the bit depth value of the image 200 to be inspected becomes smaller, the number of colors that can be expressed in each pixel of the image decreases.

The modified image generator 150 may generate the modified image 300 by reducing the bit depth value of the image to be inspected 200 to a predetermined bit value.

The predetermined bit value may be the optimal bit value derived by the user through an adversarial image detection experiment.

The bit depth value of the modified image 300 is smaller than the bit depth value of the image 200 to be inspected. For example, the bit depth value of the image to be inspected 200 may be 8 bits, and the bit depth value of the modified image 300 of the image to be inspected 200 may be 6 bits.

At this time, the color information value of each pixel of the inspection target image 200 may be an integer value between 0 and 255, but the color information value of each pixel of the modified image 300 may be an integer value between 0 and 63.

Unique integer values of the modified image 300 may also be different from those of the image to be inspected (200). That is, if the bit depth value of the inspection target image 200 is 8 bits, and the bit depth value of the modified image 300 of the inspection target image 200 is 6 bits, the unique integer value of the modified image 300 is 0 to 0. It can be an integer out of 63.

The number of pixels of the same color for each unique integer may be different from the image to be inspected (200) in the modified image (300). At this time, the sum of the number of pixels of the same color in a plurality of adjacent unique integers in the image to be inspected 200 may be the number of pixels of the same color in one unique integer.

For example, if the bit depth value of the inspection target image 200 is 8 bits and the bit depth value of the modified image 300 of the inspection target image 200 is 6 bits, the unique integer value of the inspection target image 200 The sum of the numbers of pixels of 0, 1, 2, and 3 of the same color may be the number of pixels of the same color with a unique integer value of 0 of the deformed image 300. That is, in the image 200 to be inspected, the number of same-color pixels with a unique integer value of 0 is 3, the number of same-color pixels with a unique integer value of 1 is 5, the number of pixels of the same color with a unique integer value of 2 is 1, and the number of pixels of the same color with a unique integer value of 3 is 1. If the number of pixels of the same color is 3, the number of pixels of the same color with a unique integer value of 0 of the deformed image 300 may be 12, which is the sum of all four values.

In this way, the modified image 300 has different color information values for each pixel from the image to be inspected 200, the number of unique integers is different, the number of pixels of the same color for each unique integer is different, and the same color for each unique integer. Since the pixel ratio also changes, a color information entropy value that is completely different from the color information entropy value of the original inspection target image 200 may be calculated.

That is, the information entropy calculation unit 130 may calculate the color information entropy value of the modified image 300 based on the color information values for each pixel of the modified image 300.

Specifically, the information entropy calculator 130 may derive the number of pixels of the same color, which is the number of pixels having the same unique integer as a color information value, for each unique integer in the modified image 300.

Thereafter, the information entropy calculation unit 130 divides the number of pixels of the same color of each unique integer by the number of pixels of the modified image 300 to calculate the ratio of pixels of the same color falling within the range of 0 to 1 of each unique integer. .

Finally, the information entropy calculation unit 130 calculates the color information entropy value of the transformed image 300 based on the logarithmic function application values derived by taking the logarithmic function to the ratio of the same color pixels of each unique integer of the transformed image 300. can be calculated.

Meanwhile, in the above example, the bit depth value of the inspection target image 200 is 8 bits and the bit depth value of the modified image 300 of the inspection target image 200 is 6 bits. However, the inspection target image 200 has a bit depth value of 8 bits. The difference in bit depth values between the image 200 and the modified image 300 does not necessarily have to be a 2-bit difference.

For example, when the difference in bit depth values between the inspection target image 200 and the modified image 300 is 1 bit, the hostile modified image 400 may be well detected.

That is, at this time, the modified image generator 150 may generate the modified image 300 by reducing the bit depth value of the inspection target image 200 by 1.

FIG. 4 is a diagram for explaining the ratio distribution of pixels of the same color between a normal image and an adversarially modified image, and FIG. 5 is another diagram illustrating the distribution of the pixel ratio of the same color between a normal image and an adversarially modified image.

4 and 5, the normal image 500 and each hostile modified image 400, and the modified image 300 of the normal image 500 and the modified image 300 of the hostile modified image 400. You can roughly check the ratio distribution of pixels of the same color.

In each graph, the horizontal axis may be the value obtained by dividing each unique integer for the corresponding image by the bit depth value of the image, and the vertical axis may be the number of pixels of the same color for each unique integer.

If you check the distribution of the ratio of pixels of the same color in the normal image 500 before the bit depth is reduced, you can see that the number of pixels of the same color is distributed without a unique integer between the number of pixels of the same color.

Accordingly, when the normal image 500 is the inspection target image 200, it can be confirmed that the pixel ratio distribution of the same color in the deformed image 300 is not much different from before deformation. That is, the color information entropy value of the normal image 500 before change may not be significantly different from the color information entropy value of the normal image 500 with a reduced bit depth.

On the other hand, if you check the distribution of the ratio of pixels of the same color in the adversarial transformation image 400 before the bit depth is reduced, you can see that the number of pixels of the same color is distributed as if there are unique integers with a ratio of the number of pixels of the same color in the middle. there is.

If the bit depth of the adversarial transformation image 400 is reduced, unique integers with an empty number of pixels of the same color may disappear.

For example, when the bit depth value of the adversarial deformed image 400 is 8 bits and the bit depth value of the deformed image 300 of that image is 7 bits, the same color with the unique integer value 0 of the adversarial deformed image 400 If the number of pixels is 2000, the number of same-color pixels with unique integer value 1 is 1, the number of same-color pixels with unique integer value 2 is 1900, and the number of same-color pixels with unique integer value 3 is 2, then the unique integer value of the transformed image 300 is The number of same-color pixels of 0 may be 2001, and the number of same-color pixels of the modified image 300 with a unique integer value of 1 may be 1902.

That is, in the case of the adversarial transformation image 400 before change, the number of pixels of the same color with unique integer values 1 and 3 is too small, so when checking the distribution of the ratio of pixels of the same color, there are unique integers with an empty number of pixels of the same color in the middle. In the case of the adversarially modified image 400 after modification, the number of pixels of the same color in the unique integer is combined with the number of pixels of the same color in other adjacent unique integers, so the unique integers with the number of pixels in the same color may disappear in the middle.

Since this excessively small number of pixels of the same color disappears, the adversarial deformation image 400 with a reduced bit depth may have a color information entropy value that is significantly different from the adversarial deformation image 400 before the bit depth reduction.

The adversarial modified image determination unit 140 may calculate the difference between the color information entropy value of the inspection target image 200 and the color information entropy value of the modified image 300. At this time, the difference may be calculated by subtracting the color information entropy value of the inspection target image 200 from the color information entropy value of the modified image 300.

If the difference between the color information entropy value of the inspection target image 200 and the color information entropy value of the modified image 300 is greater than or equal to the reference difference value, the hostile modified image determination unit 140 converts the inspected image 200 into the hostile modified image ( 400).

Figure 6 is a diagram illustrating a process of separately classifying an inspection target image determined to be an adversarial modified image according to an embodiment.

Referring to FIG. 6, the bit depth of the image to be inspected 200 is reduced to generate a deformed image 300, and the color information entropy of the generated deformed image 300 and the image to be inspected 200 are compared to determine adversarial deformation. The process of detecting the image 400 can be confirmed.

In addition, it can be confirmed that among the inspection target images 200, the image determined to be the hostile modified image 400 is not input to the image analysis device 600.

The hostile deformation image determination unit 140 may store the image classified as the adversarial deformation image 400 separately from the remaining inspection target images 200 to prevent the adversarial deformation image 400 from being input to the image analysis device 600. there is.

Figure 7 is a diagram for explaining a reference difference value according to an embodiment.

The reference difference value is the difference between the color information entropy value of the inspection target image 200 and the color information entropy value of the modified image 300, which is the standard for determining whether the inspection target image 200 is a hostile modified image 400. You can.

Referring to FIG. 7 , it can be seen that the reference difference value can be determined based on the difference between the color information entropy value of the plurality of normal images (Legitimate) and the color information entropy value of the deformed image 300.

Specifically, the differences between the color information entropy value calculated based on a plurality of images that have already been determined as normal and the color information entropy value of the modified image 300 are calculated and listed in order of size, according to the classification accuracy desired by the user. The difference value in the top ratio specified by the user can be set as the standard difference value.

Referring to the graph shown, the difference between the color information entropy value of the normal images 500 and the color information entropy value of the modified image 300 is not significantly different from the color information entropy value of any specific normal image 500. It can be seen that the difference in the color information entropy value of the modified image 300 can be determined as a reference difference value (threshold).

The input unit 110 can receive a plurality of normal images 500 as input. Since the plurality of input normal images 500 are different images, the color information entropy of each may be different.

The information entropy calculation unit 130 may calculate the color information entropy value of the normal image 500 based on the color information values for each pixel of the input normal image 500.

The modified image generator 150 may generate a modified normal image by reducing the bit depth value of the normal image 500 to a predetermined bit value.

The information entropy calculation unit 130 may calculate the color information entropy value of the deformed normal image based on the color information values for each pixel of the deformed normal image.

The reference difference value determination unit 160 may calculate the difference between the color information entropy value of the normal image 500 and the color information entropy value of the deformed normal image.

When the difference values of the color information entropy values of the plurality of normal images 500 are arranged in order of size, the reference difference value determination unit 160 determines the color information entropy value of the normal image 500 corresponding to the pre-specified higher order number. The difference can be determined as a reference difference value (threshold).

Any type of adversarial deformation image (FGSM(0.01), FGSM(0.03), BIM(0.01), BIM(0.03), CW_Next, CW_LL) (400) is an adversarial deformation image with reduced color information entropy value and bit depth value. It can be seen that the difference in the color information entropy value of (400) is generally larger than the standard difference value (threshold).

At least one component may be added or deleted in response to the performance of the components described above. Additionally, it will be easily understood by those skilled in the art that the mutual positions of the components may be changed in response to the performance or structure of the system.

Figure 8 is a flowchart of an adversarial modified image detection method according to one embodiment. This is only a preferred embodiment for achieving the purpose of the present invention, and of course, some components may be added or deleted as needed.

Referring to FIG. 8, the input unit 110 may receive an inspection target image 200 (1001).

The pixel color information extraction unit 120 extracts a color information value of the inspection target image 200, which is one integer value among unique integers that are integers within the bit depth value range of the inspection target image 200, for each pixel of the inspection target image 200. Can be extracted (1002).

The modified image generator 150 may generate the modified image 300 by reducing the bit depth value of the inspection target image 200 to a predetermined bit value (1003).

The pixel color information extraction unit 120 may extract a color information value of the transformed image 300, which is one integer value among unique integers of the transformed image 300, for each pixel of the transformed image 300 (1004).

The information entropy calculation unit 130 may calculate the color information entropy value of the image to be inspected 200 based on the color information values for each pixel of the image to be inspected 200 (1005). Additionally, the information entropy calculation unit 130 may calculate the color information entropy value of the modified image 300 based on the color information values for each pixel of the modified image 300.

At this time, the hostile modified image determination unit 140 may calculate the difference between the color information entropy value of the inspection target image 200 and the color information entropy value of the modified image 300.

The adversarial modified image determination unit 140 may determine whether the difference between the color information entropy value of the inspection target image 200 and the color information entropy value of the modified image 300 is greater than or equal to the reference difference value (1006).

If the difference between the color information entropy value of the inspection target image 200 and the color information entropy value of the modified image 300 is less than the reference difference value ('No' in 1006), the hostile modified image determination unit 140 determines the inspection target image 300. (200) can be determined to be a normal image (500) (1007).

If the difference between the color information entropy value of the inspection target image 200 and the color information entropy value of the modified image 300 is greater than or equal to the reference difference value ('Yes' in 1006), the hostile modified image determination unit 140 determines the inspection target image 300. (200) can be judged to be a hostile transformation image (400) (1008).

In order to verify the performance of the adversarially modified image detection method according to an embodiment of the present invention, an experiment was conducted to detect hostilely modified images among a plurality of experimental images.

FIG. 9 is a graph showing the average color information entropy of a normal image and an adversarial modified image according to an embodiment, and FIG. 10 is a graph showing classification accuracy according to the bit depth value of the modified image.

Referring to FIG. 9, in the case of images (Original) before reducing the bit depth, the average color information entropy value of the normal image (Legitimate) 500 is the adversarial transformation image (FGSM (0.01), FGSM (0.03), It can be seen that it is relatively smaller than the average color information entropy of BIM (0.01), BIM (0.03), CW_Next, CW_LL) (400). Therefore, simply the size of the color information entropy value of an inspection target image 200 can be a criterion for determining whether the image is an adversarial modified image 400.

Meanwhile, comparing the average color information entropy value of the images before reducing the bit depth (original) and the images with reduced bit depth (bit_depth=7, bit_depth=6, bit_depth=5), the color information entropy value before reducing the bit depth is It can be seen that the average color information entropy value of the image with reduced bit depth is smaller than that of the image.

At this time, the bit depth of the adversarial deformed image 400 and the adversarial deformed image 400 is greater than the difference in the average color information entropy value between the legitimate image 500 and the image with the bit depth of the normal image 500 reduced. It can be seen that the difference in the average color information entropy value between images with reduced is larger.

Through this, it can be determined whether the inspection target image 200 is an adversarial modified image 400 based on the difference between the color information entropy value of the inspection target image 200 and the color information entropy value of the modified image 300. can confirm.

Referring to FIG. 10, the detection rate for the hostile modified image 400 can be confirmed depending on how much the bit depth value of the inspection target image 200 with a bit depth value of 8 bits is reduced.

When the inspection target image 200 has a bit depth value of 8 bits, it can be confirmed that reducing the inspection target image 200 to 7 bits to create the modified image 300 (bit_depth=7) has the highest detection rate. there is.

Therefore, when generating the modified image 300, it can be confirmed that generating the modified image 200 by reducing the bit depth value of the image 200 to be inspected by 1 provides the most optimal classification performance.

As described above, the disclosed embodiments have been described with reference to the attached drawings. A person skilled in the art to which the present invention pertains will understand that the present invention can be practiced in forms different from the disclosed embodiments without changing the technical idea or essential features of the present invention. The disclosed embodiments are illustrative and should not be construed as limiting.

100: Adversarial deformed image detection system
110: input unit
120: Pixel color information extraction unit
130: Information entropy calculation unit
140: Adversarial transformation image decision unit
150: Deformed image generation unit
160: Reference difference value determination unit
200: Image to be inspected
300: Deformed image
400: Adversarial Transformation Image
500: Normal image
600: Image analysis device

Claims (10)

In an adversarial modified image detection method performed by an adversarial modified image detection system that determines whether the image to be inspected is an adversarial modified image created by adding noise to a normal image,
Receiving the image to be inspected as input by an input unit provided in the adversarial deformed image detection system;
By the pixel color information extractor provided in the adversarial deformed image detection system, a color information value that is an integer value of one of unique integers that are integers within the bit depth value range of the image to be inspected for each pixel of the image to be inspected. Extracting;
calculating a color information entropy value of the image to be inspected based on color information values for each pixel of the image to be inspected, by an information entropy calculation unit provided in the adversarial deformed image detection system;
Comparing the color information entropy value of the image to be inspected with a reference entropy value by an adversarial modified image determination unit provided in the hostile modified image detection system;
classifying the inspection target image as the adversarial modified image if the color information entropy value of the inspection target image is greater than or equal to the reference entropy value, by the hostile modified image determination unit;
generating a modified image by reducing the bit depth value of the image to be inspected to a predetermined bit value by a modified image generator provided in the hostile modified image detection system;
calculating a color information entropy value of the transformed image based on the color information values for each pixel of the transformed image, by the information entropy calculation unit;
calculating a difference between a color information entropy value of the image to be inspected and a color information entropy value of the modified image by an adversarial modified image determination unit provided in the hostile modified image detection system; and
By the hostile modified image determination unit, if the difference between the color information entropy value of the inspection target image and the color information entropy value of the modified image is greater than or equal to a reference difference value, classifying the inspection target image as a hostile modified image. An adversarial morphed image detection method. delete According to paragraph 1,
The step of calculating the color information entropy value of the image to be inspected is,
By the information entropy calculation unit, the number of identical color pixels, which is the number of pixels having the same unique integer as a color information value, is derived for each unique integer in the image to be inspected, and the number of identical color pixels of each unique integer is calculated as Calculating a ratio of pixels of the same color falling within the range of 0 to 1 for each unique integer by dividing by the number of pixels in the image to be inspected; and
Calculating, by the information entropy calculation unit, a color information entropy value of the image to be inspected based on logarithmic function application values derived by taking a logarithmic function to the same color pixel ratio of each unique integer of the image to be inspected. Including, adversarial modified image detection method. delete delete According to paragraph 1,
The step of calculating the color information entropy value of the modified image is,
By the information entropy calculator, the number of same-color pixels, which is the number of pixels having the same unique integer as a color information value, is derived for each unique integer in the transformed image, and the number of same-color pixels of each unique integer is calculated as Calculating a ratio of pixels of the same color falling within the range of 0 to 1 for each unique integer by dividing by the number of pixels in the transformed image; and
Comprising the step of calculating, by the information entropy calculation unit, a color information entropy value of the transformed image based on logarithmic function application values derived by taking a logarithmic function for the same color pixel ratio of each unique integer of the transformed image. ,Adversarial modified image detection method. According to paragraph 1,
receiving a plurality of normal images as input through the input unit;
calculating a color information entropy value of the normal image based on color information values for each pixel of the normal image, by the information entropy calculation unit;
generating a modified normal image by reducing the bit depth value of the normal image to a predetermined bit value, by the modified image generator;
calculating a color information entropy value of the deformed normal image based on the color information values for each pixel of the deformed normal image, by the information entropy calculation unit;
calculating a difference between a color information entropy value of the normal image and a color information entropy value of the modified normal image, by a reference difference value determination unit provided in the hostile modified image detection system; and
When the difference values of the color information entropy values of the plurality of normal images are arranged in order of size by the reference difference value determination unit provided in the adversarial deformed image detection system, the color information of the normal image corresponding to the predetermined upper order number An adversarial modified image detection method further comprising determining the difference in entropy values as the reference difference value. According to paragraph 1,
The step of generating the modified image is,
An adversarial modified image detection method comprising generating the modified image by reducing the bit depth value of the image to be inspected by 1, by the modified image generator. According to paragraph 1,
The hostile modified image further comprising the step of storing, by the hostile modified image determination unit, the image classified as the hostile modified image separately from the remaining images to be inspected so that the hostile modified image is not input to the image analysis device. Detection method. A computer program stored in a computer-readable recording medium to execute the method of detecting an adversarial modified image according to any one of claims 1, 3, and 6 to 9.