KR102486795B1 - Method and Apparatus for Data Augmentation in Frequency Domain for High Performance Training in Deep Learning - Google Patents

Abstract

A method and apparatus for augmenting data in the frequency domain for improving performance of deep learning are presented. The data augmentation method in the frequency domain for improving the performance of deep learning proposed in the present invention includes performing filtering in the frequency domain by applying a Fourier transform to remove bias in data, and generating a new image using the filtered image. and performing learning based on the generated image.

Description

Method and Apparatus for Data Augmentation in Frequency Domain for High Performance Training in Deep Learning}

The present invention relates to a method and apparatus for augmenting data in a frequency domain for improving performance of deep learning.

The imbalance of data when training multi-image classification models has been an important topic so far in improving classification performance. If the distribution of specific data is excessively biased compared to other data, the trained model shows high accuracy for the biased data with respect to the test data, but low accuracy for the other data. To solve this problem, various studies have been conducted to generalize the model. Typically, bias is reduced through dropout, batch normalization, and data augmentation techniques.

First, dropout is a technique for randomly inactivating neurons in a specific layer while the model is being trained. Through this, an ensemble effect can often be obtained, and generalization of the model can be achieved.

Next, data augmentation learns the model after going through processes such as enlargement, reduction, rotation, horizontal inversion, and vertical inversion of the input image. Through this, the bias of the data is reduced and the accuracy is increased.

However, there are cases where the above method does not solve the problem. To solve this phenomenon, we propose a method of obtaining a new image through filtering preprocessing in the frequency domain for an image with a small number of data. In addition, through various experiments, we find a filter that best improves the classification accuracy performance of the given data.

A technical problem to be achieved by the present invention is to provide a method and apparatus for creating a new image by applying a ButterWorth cut-off filter to a specific image in a frequency space. Through this, we propose a method and device to prevent overfitting by solving the imbalance problem of existing data.

In one aspect, the data augmentation method in the frequency domain for improving the performance of deep learning proposed in the present invention includes the steps of performing filtering in the frequency domain by applying a Fourier transform to remove bias in data, and generating the filtered image. It includes generating a new image using the image and performing learning based on the generated image.

In the step of filtering in the frequency domain by applying Fourier transform to remove data bias, preprocessing is performed to obtain a filtered image through pixel-by-pixel multiplication using a predetermined filter on the image to which the Fourier transform is applied.

The step of performing filtering in the frequency domain by applying a Fourier transform to remove the bias of the data is to move the center of the transformed image by applying a 2D discrete Fourier transform to the image, and to filter the image transformed in the frequency domain and filter. Perform pixel-by-pixel multiplication for

In the step of generating a new image using the filtered image, a second-order inverse discrete Fourier transform is applied to convert the filtered image from the frequency domain back to the spatial domain, and the center of the image converted to the spatial domain is moved.

In another aspect, the data augmentation apparatus in the frequency domain for improving the performance of deep learning proposed in the present invention includes a filtering unit that performs filtering in the frequency domain by applying a Fourier transform to remove data bias, filtering It includes an image generation unit for generating a new image using the generated image and a learning unit for performing learning based on the generated image.

According to embodiments of the present invention, a new image may be obtained through pre-processing of filtering in the frequency domain for an image having a small number of data. In addition, through this, overfitting can be prevented by solving the imbalance problem of existing data.

1 is a flowchart illustrating a data augmentation method in a frequency domain for improving performance of deep learning according to an embodiment of the present invention.
2 is a diagram showing a filtered image according to an embodiment of the present invention.
3 is a diagram showing the configuration of a data augmentation device in the frequency domain for deep learning performance improvement according to an embodiment of the present invention.
4 is a diagram showing examples of skin cancer and data distribution for an experiment according to an embodiment of the present invention.
5 is a diagram showing learning results for data to which an augmentation method according to an embodiment of the present invention is applied.
6 is a diagram showing an error rate for test data for each data according to an embodiment of the present invention.
7 is a diagram comparing a learning result using an augmentation technique according to an embodiment of the present invention with a learning result according to the prior art.
8 is a diagram comparing a learning result using an augmentation technique according to another embodiment of the present invention with a learning result according to the prior art.
9 is a diagram showing distribution results for data augmentation ratios according to an embodiment of the present invention.
10 is a diagram showing a result of applying frequency domain filtering and data augmentation method according to an embodiment of the present invention.

The present invention proposes a method of creating a new image by applying a ButterWorth cut-off filter to a specific image in the frequency space. This will solve the imbalance problem of the existing data and prevent overfitting. In addition, in order to assume that the medical image is a gray scale image, the image is converted to gray scale, followed by Fourier transform and filtering in the frequency domain. For the experiments, VGG16 was used with fine adjustment. As a result, by applying the two methods together, the error rate of the smallest data with only 95 items was reduced by about 20%. Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a flowchart illustrating a data augmentation method in a frequency domain for improving performance of deep learning according to an embodiment of the present invention.

The proposed method for augmenting data in the frequency domain for improving performance of deep learning includes performing filtering in the frequency domain by applying Fourier transform to remove data bias (110), and generating a new image using the filtered image. A step 120 of generating and a step 130 of performing learning based on the generated image are included.

In step 110, filtering in the frequency domain is performed by applying a Fourier transform to remove data bias. Preprocessing is performed to obtain a filtered image through pixel-by-pixel multiplication using a predetermined filter on the image to which the Fourier transform is applied. The 2D discrete Fourier transform is applied to the image, the center of the transformed image is moved, and pixel-by-pixel multiplication is performed on the transformed image and filter in the frequency domain.

In step 120, a new image is generated using the filtered image. In order to convert the image filtered in the frequency domain back to the spatial domain, a second-order inverse discrete Fourier transform is applied, and the center of the image transformed to the spatial domain is shifted.

In step 130, learning is performed based on the generated image.

2 is a diagram showing a filtered image according to an embodiment of the present invention.

Frequency domain filtering is a pre-processing that obtains a filtered image by multiplying a specific filter per pixel on an image to which Fourier transform is applied. The proposed preprocessing process is as follows.

First, a 2D discrete Fourier transform is applied to the image 210. Then, the center of the transformed image is shifted. Pixel-by-pixel multiplication is performed on the image 220 transformed in the frequency domain and the filter 230. Then, a second-order inverse discrete Fourier transform is applied to convert it back to the spatial domain. Finally, the center of the image 240 returned to the spatial domain is moved.

In FIG. 2, a filtered image is shown by applying a ButterWorth blocking filter with width = 32, range = 40, and n = 10.

At this time, a Butterworth cut-off filter was applied among ideal cut-off filters and Gaussian cut-off filters. Comparing the original image of FIG. 2 with the filtered image, it can be seen that there is not such a big difference in identification with the human eye. However, the overall pixel value changed little by little.

3 is a diagram showing the configuration of a data augmentation device in the frequency domain for deep learning performance improvement according to an embodiment of the present invention.

The proposed apparatus for augmenting data in the frequency domain for improving performance of deep learning includes a filtering unit 310, an image generator 320, and a learning unit 330.

The filtering unit 310 performs filtering in the frequency domain by applying a Fourier transform to remove data bias. Preprocessing is performed to obtain a filtered image through pixel-by-pixel multiplication using a predetermined filter on the image to which the Fourier transform is applied. The 2D discrete Fourier transform is applied to the image, the center of the transformed image is moved, and pixel-by-pixel multiplication is performed on the transformed image and filter in the frequency domain.

The image generator 320 generates a new image using the filtered image. In order to convert the image filtered in the frequency domain back to the spatial domain, a second-order inverse discrete Fourier transform is applied, and the center of the image transformed to the spatial domain is shifted.

The learning unit 330 performs learning based on the generated image.

4 is a diagram showing examples of skin cancer and data distribution for an experiment according to an embodiment of the present invention.

The experiment process is as follows. For the fixed experimental variable, learning was repeated for 200 epochs, and 'Adam' was selected as the optimizer for optimization. First, the classification performance of the existing data and the data to which the existing data augmentation method is applied is checked. Next, using the filtering preprocessing described above, a filter with the highest classification performance is selected through experiments on various filters. Finally, even after applying both the existing data augmentation method and the data augmentation method proposed in the present invention, a filter with the highest classification performance is selected. Finally, the classification performance of the data obtained through the selected filters is compared with the performance of the existing data augmentation technique.

Data were collected through a big data site called Kaggle, and seven types of skin cancers had different distributions, so they were selected because they were appropriate for the experiment. An example of skin cancer and data distribution are shown in FIG. 4 .

Fig. 4(a) shows a diagram in which 5 samples of 7 types of skin cancer were respectively sampled. 4(b) shows the number of data for each skin cancer. From left data: Melanocytic nevi (nv): 5498, Benign keratosis-like lesions (bkl): 907, Melanoma (mel): 892, Basel cell carcinoma (bcc): 430, Actinic keratoses (akiec): 262 , Vascular lesions (vasc): 117, Dermatofibroma (df): 95.

5 is a diagram showing learning results for data to which an augmentation method according to an embodiment of the present invention is applied.

First, the result of learning about the data without any change and the data to which the existing data augmentation method is applied is checked. At this time, rotation between 0 and 10 degrees, scaling within 10% centered on the center, or horizontal/vertical movement by 0.1 times the total pixel size were applied to the input image for each learning by data augmentation.

The results are shown in FIG. 5 . Figure 5 (a) shows the accuracy for the existing data, and Figure 5 (b) shows the change in loss for the basic data. Here, training data accuracy = 97%, verification data accuracy = 72%, and test data accuracy = 73.6%.

FIG. 5(c) shows the accuracy of data obtained by applying the proposed augmentation method to existing data, and FIG. 5(d) shows changes in loss for data obtained by applying the proposed augmentation method to existing data. Here, training data accuracy = 74%, validation data accuracy = 71%, and test data accuracy = 74.2%.

If you check the trend of the accuracy and loss of the existing data, you can see that the accuracy of the training data is converging to 100% and the loss to 0. However, in the case of the verification data, it can be seen that the accuracy diverges around 72%, and the loss increases again around 20 epochs. On the other hand, when the existing data augmentation method is applied, the accuracy is about 85% and the loss converges to around 0.3 for the training data. Judging only by the performance of the training data, it can be judged that it will have worse performance than the existing data, but looking at the loss of the verification data, it can be seen that the rate of increase again is reduced compared to the existing data. This can be seen as a decrease in the bias of the data compared to the existing data.

6 is a diagram showing an error rate for test data for each data according to an embodiment of the present invention.

Referring to FIG. 6 , it can be seen that the error rate for the bcc, df, and nv data of the data 620 to which the augmentation method is applied is reduced compared to the existing data 610 . However, in the case of bkl, mel, and vasc data, it can be seen that the error rate increases or does not change significantly. Since the existing data augmentation method is incomplete, it can be seen that a new data augmentation method is needed to compensate for this.

7 is a diagram comparing a learning result using an augmentation technique according to an embodiment of the present invention with a learning result according to the prior art.

7 shows the change trend of verification data loss after data augmentation at a ratio of 4:1 applying a ButterWorth cut-off filter at n = 2, 5, and 10 from the top. At this time, according to each n, the left 3 columns are the result of learning after applying filtering by setting the filtering range to 20, 30, and 40 for the existing data and setting an appropriate bandwidth value for one range. The third column on the right is the result of learning by applying both data filtering and existing data augmentation techniques. This is summarized in Table 1 below.

<Table 1>

8 is a diagram comparing a learning result using an augmentation technique according to another embodiment of the present invention with a learning result according to the prior art.

8 shows the change trend of verification data loss after data augmentation at a ratio of 2:1 by applying a ButterWorth cut-off filter at n = 2, 5, and 10 from the top. At this time, according to each n, the left 3 columns are the result of learning after applying filtering by setting the filtering range to 20, 30, and 40 for the existing data and setting an appropriate bandwidth value for one range. The third column on the right is the result of learning by applying both data filtering and existing data augmentation techniques. This is summarized in Table 2 below.

<Table 2>

The following is a process of applying filtering by dividing the data augmentation ratio into 4:1 and 2:1 ratios when using the data augmentation technique using frequency domain filtering.

9 is a diagram showing distribution results for data augmentation ratios according to an embodiment of the present invention.

9(a) is a distribution when data augmentation is performed with a data augmentation ratio of 4:1, and FIG. 9(b) is a distribution when data augmentation is performed with a data augmentation ratio of 2:1. Results are shown in FIG. 7 .

In all graphs, it can be seen that the loss of verification data converges more stably as the size of the inner pass filter increases. In this case, when Fourier transform is applied to the image and the center is shifted, a phenomenon occurs in which all information of the image is shifted to the center. At this time, the center of the image is called a dc term. Accordingly, it can be seen that the closer to the center of the transformed image, the more information of the image exists. At this time, the size of the inner pass filter is a pass filter that draws a circle centered on the image, and it can be seen that as the bandwidth increases, more image information is preserved. For this reason, the loss of verification data converges better in a filter with a larger inner pass filter than in a filter with a small size.

At this time, for the same reason, the larger the size of the blocking filter is, the more information of the image is not blocked, so the larger the size of the blocking filter is, the better the loss of verification data converges.

However, while the accuracy on the training data is still almost 100%, the accuracy on the validation data is generally between 87% and 91%. This means that the bias in the data is still not removed.

Finally, the result of applying the existing data augmentation method and the method proposed in the present invention together for the same data augmentation ratio is confirmed. Compared to the case where only the existing data augmentation method was applied, the accuracy of the training data and verification data increased by about 10%, and it shows the result of convergence around 0.6 instead of convergence around 0.9.

If the data augmented at a ratio of 2:1 also increases the accuracy of the training data and verification data by about 10%, it can be seen that the loss of the verification data converges around 0.3 to 0.4.

10 is a diagram showing a result of applying frequency domain filtering and data augmentation method according to an embodiment of the present invention.

10(a) is a result of applying only frequency domain filtering, and FIG. 10(b) is a result of applying both frequency domain filtering and an existing data augmentation method.

In the present invention, in order to remove the bias of data, a method of generating a new image through filtering in the frequency domain rather than the existing data augmentation method was presented, and the process of learning through VGG16 based on this was presented. In particular, the present invention proposes adding and using a new image to which filtering in the frequency domain is applied before using the existing data augmentation method. Although it did not show a significant effect on some skin cancer data, it was found that the wrong answer rate decreased by about 20% in the case of skin cancer, which had the smallest number of data. Through this, it can be seen that even small data can have better performance if it is learned using other data augmentation methods. In the present invention, learning is performed using gray scale images, but learning using color images is also possible.

The devices described above may be implemented as hardware components, software components, and/or a combination of hardware components and software components. For example, devices and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable array (FPA), It may be implemented using one or more general purpose or special purpose computers, such as a programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions. A processing device may run an operating system (OS) and one or more software applications running on the operating system. A processing device may also access, store, manipulate, process, and generate data in response to execution of software. For convenience of understanding, there are cases in which one processing device is used, but those skilled in the art will understand that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that it can include. For example, a processing device may include a plurality of processors or a processor and a controller. Other processing configurations are also possible, such as parallel processors.

Software may include a computer program, code, instructions, or a combination of one or more of the foregoing, which configures a processing device to operate as desired or processes independently or collectively. You can command the device. Software and/or data may be any tangible machine, component, physical device, virtual equipment, computer storage medium or device, intended to be interpreted by or provide instructions or data to a processing device. can be embodied in Software may be distributed on networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer readable media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer readable medium. The computer readable medium may include program instructions, data files, data structures, etc. alone or in combination. Program commands recorded on the medium may be specially designed and configured for the embodiment or may be known and usable to those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. - includes hardware devices specially configured to store and execute program instructions, such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include high-level language codes that can be executed by a computer using an interpreter, as well as machine language codes such as those produced by a compiler.

As described above, although the embodiments have been described with limited examples and drawings, those skilled in the art can make various modifications and variations from the above description. For example, the described techniques may be performed in an order different from the method described, and/or components of the described system, structure, device, circuit, etc. may be combined or combined in a different form than the method described, or other components may be used. Or even if it is replaced or substituted by equivalents, appropriate results can be achieved.

Therefore, other implementations, other embodiments, and equivalents of the claims are within the scope of the following claims.

Claims (8)

performing filtering in the frequency domain by applying a Fourier transform to remove data bias;
generating a new image using the filtered image; and
Performing learning based on the generated image
including,
Performing filtering in the frequency domain by applying a Fourier transform to remove the bias of the data,
In order to prevent overfitting by solving the imbalance problem of data for an image, the center of the image to which the 2D Fourier transform is applied in the frequency space is shifted, and a predetermined filtering range for the center-shifted image and a bandwidth for the range After augmenting the data using a ButterWorth cut-off filter, preprocessing is performed to obtain a filtered image through pixel-by-pixel multiplication
Data augmentation method. delete delete According to claim 1,
The step of generating a new image using the filtered image is,
In order to convert the image filtered in the frequency domain back to the spatial domain, a second-order inverse discrete Fourier transform is applied, and the center of the image transformed to the spatial domain is shifted.
Data augmentation method. a filtering unit performing filtering in a frequency domain by applying a Fourier transform to remove data bias;
An image generator for generating a new image using the filtered image; and
A learning unit that performs learning based on the generated image
including,
The filtering unit,
In order to prevent overfitting by solving the imbalance problem of data for an image, the center of the image to which the 2D Fourier transform is applied in the frequency space is shifted, and a predetermined filtering range for the center-shifted image and a bandwidth for the range After augmenting the data using a ButterWorth cut-off filter, preprocessing is performed to obtain a filtered image through pixel-by-pixel multiplication
Data Augmentation Device. delete delete According to claim 5,
video generator,
In order to convert the image filtered in the frequency domain back to the spatial domain, a second-order inverse discrete Fourier transform is applied, and the center of the image transformed to the spatial domain is shifted.
Data Augmentation Device.

Images