KR20210029110A - Method and apparatus for few-shot image classification based on deep learning - Google Patents

Abstract

According to an aspect of the present application, a deep learning-based fractional shot image classification method includes the steps of: a primary learning step of performing data augmentation processing on an input image according to a self-mixing technique which generates a new input image by replacing a part of the input image with another part of the input image, performing feature extraction on the input images on which data augmentation has been performed, learning the extracted features based on deep learning, and creating a classification model which classifies the input image; a secondary learning step of creating a new auxiliary classifier from a main classifier included in the classification model and allowing independently output results of each classifier to be shared with other classifiers according to a self-distillation technique; and a step of outputting a classification result by inputting the new input image to the image classification model that has undergone the primary and secondary learning steps, thereby providing a high level of classification accuracy.

Description

Deep learning-based prime shot image classification apparatus and method {METHOD AND APPARATUS FOR FEW-SHOT IMAGE CLASSIFICATION BASED ON DEEP LEARNING}

The present invention relates to an apparatus and method capable of classifying a small number of shot images based on deep learning.

Recently, image processing technology has been greatly advanced due to the advent of big data and the development of deep learning technology in the field of artificial intelligence accompanied by the development of computer hardware, and based on this, technology for automatically classifying images in various fields is developing. . Unlike image processing methods based on existing algorithms or image processing methods based on statistics, image processing technology based on deep learning enables computers to automatically classify even complex images that are difficult for humans to classify.

However, in order to achieve high-performance classification technology in image processing, a large number of learning data labeled with correct answers are required. However, many industries and academic fields do not have enough data for such learning. It is a situation that is difficult in terms of time and cost because the work to process is required.

Recently, research on a fractional shot learning technique has been attracting attention in order to improve classification performance in image processing. The prime shot learning technique is a study aiming to properly classify the input few images when a new image that has not been seen before is input to a model that has been trained with a large number of data. Even if the model is trained with a large number of data, hundreds or more of training data are required to classify a new category, but the prime shot learning technique is a technology that enables this process with only a few images.

The fractional shot learning technique largely takes the same learning method as the general image classification method such as feature extraction and classification, but in detail, it can be classified into a distance-based learning method, an optimization method, a data augmentation method, and other methods.

The research on image classification in the prior art has been conducted by extracting patterns and features for images corresponding to each class based on a large number of big data, and recently, a technical improvement that can improve accuracy while reducing computation. The development of a classification model focused on or specific to a specific domain has been studied.

As described above, image classification technology has been developed, but it has been pointed out as a limitation of deep learning based image classification technology that the related technology cannot be applied unless sufficient amount of data for training a classification model is secured.

In order to overcome these limitations, studies on image classification methods based on fractional shot learning have been conducted in recent years, and due to these studies, studies on fractional shot learning systems have been able to achieve relatively high performance within a specific data set. Technology development took place.

However, when a data set completely different from the data set used for training is applied to the performance evaluation stage of the fractional shot learning system, the phenomenon that the classification performance is lowered was reported in one study. Another study found that when deep learning models are trained, they tend to memorize input training data. This is linked to the overfitting problem of deep learning models, and this learning data memorization can be considered to have a great effect on hindering the generalization performance of deep learning.

Eventually, previous studies showed that there was a limit to achieving high generalization performance in fractional shot learning, and showing a slightly improved generalization performance with a new learning method different from the previous one began to emerge as an important task for subsequent studies. .

As existing minority shot learning models focus on improving the accuracy of classification using only the deep learning-based image processing process, it is judged that the recognition accuracy for completely different data sets that may occur in reality has not been considered.

Republic of Korea Patent Publication No. 10-1888647 (title of the invention: image classification apparatus and method)

The present invention is to solve the above-described problem, by using a self-augmentation technique that makes various changes to the data distribution and output distribution of an input image, and a fine-tuning technique called regional feature learning, deep learning-based classifier learning is possible with only a small number of data. An object of the present invention is to provide a method and apparatus for classifying a small number of shot images that provide a possible and high level of classification accuracy.

However, the technical problem to be achieved by the present embodiment is not limited to the technical problem as described above, and other technical problems may exist.

As a technical means for solving the above technical problem, the deep learning-based fractional shot image classification method according to the first aspect of the present disclosure generates a new input image by replacing a part of the input image with another part of the input image. Perform data augmentation processing on the input image according to the self-mixing technique, perform feature extraction on the input images with data augmentation, learn the extracted features based on deep learning learning, and classify the input images. A first learning step of generating a classification model; The second learning step and the first learning step and 2 in which a new auxiliary classifier is created from the main classifier for inclusion in the classification model, and the independently output results of each classifier are shared with other classifiers according to the self-distillation technique. And outputting a classification result by inputting a new input image to the image classification model that has undergone the primary learning step.

In addition, an apparatus for classifying a fractional shot image based on deep learning according to a second aspect of the present disclosure includes: a communication module; A memory in which a fractional shot image classification program is stored; A processor for executing a program stored in the memory, wherein the fractional shot image classification program replaces a part of the input image with another part of the input image to generate a new input image. A first learning step of performing augmentation processing, performing feature extraction on input images with data augmentation, learning the extracted features based on deep learning learning, and generating a classification model for classifying input images; and To include in the classification model, a new sub-classifier is generated from the main classifier, and the independently output result of each classifier is shared with other classifiers according to the self-distillation technique. And the fractional shot image classification program inputs a new input image to the image classification model and outputs a classification result of the new input image.

According to any one of the above-described problem solving means of the present application, since a data augmentation technique called self-mixing is used, a phenomenon in which the deep learning technology memorizes the learning data can be prevented. In addition, a change can be applied to the output distribution of the model through a self-distillation technique based on an auxiliary classifier. This allows the system to more sensitively acquire various pattern information of the data rather than memorizing the learning data, thereby improving the accuracy of correctly recognizing objects in the image even for a small number of new images.

With this effect, it is possible to construct an image classification model capable of solving the data shortage phenomenon experienced in industry and academia without having to go through the cumbersome task of manually labeling the data set.

1 is a block diagram illustrating an apparatus for classifying a fractional shot image based on deep learning according to an embodiment of the present invention.
2 is a flowchart illustrating a method for classifying a minority shot image according to another embodiment of the present invention.
3 is a diagram for explaining a self-augmentation technique used in a method for classifying a minority shot image according to an exemplary embodiment of the present invention.
4 is a diagram illustrating a feature extraction process used in a method for classifying a prime number shot image according to an exemplary embodiment of the present invention.
5 is a diagram illustrating a self-distillation technique used in a method for classifying a fractional shot image according to an exemplary embodiment of the present invention.

Hereinafter, embodiments of the present application will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art may easily implement the present application. However, the present application may be implemented in various different forms and is not limited to the embodiments described herein. In addition, in the drawings, parts irrelevant to the description are omitted in order to clearly describe the present application, and similar reference numerals are attached to similar parts throughout the specification.

Throughout this specification, when a part is said to be "connected" with another part, this includes not only the case that it is "directly connected", but also the case that it is "electrically connected" with another element interposed therebetween. do.

Throughout this specification, when a member is positioned “on” another member, this includes not only the case where a member is in contact with the other member, but also the case where another member exists between the two members.

1 is a block diagram illustrating an apparatus for classifying a prime number shot image based on deep learning according to an embodiment of the present invention.

It may include a fractional shot image classification apparatus 100, a communication module 110, a memory 120, a processor 130, a database 140, and various input/output modules 150 according to the present invention.

The communication module 110 may transmit and receive various types of data through a wired/wireless communication network connection with the minority shot image classification apparatus 100 and other computing devices or servers. In particular, the communication module 110 may receive image data required for learning, receive a new input image to be classified as an image, or transmit a classification result for a new input image.

A prime shot image classification program is stored in the memory 120. The fractional shot image classification program performs data augmentation processing on the input image according to a self-mixing technique that generates a new input image by replacing a part of the input image with another part of the corresponding input image, and performs data augmentation on the input images. A new auxiliary classifier from the main classifier is used in the primary learning step of generating a classification model for classifying the input image by performing feature extraction, learning the extracted features based on deep learning learning, and inclusion in the classification model. It includes an image classification model generated through a secondary learning step in which the generated and independently outputted results of each classifier are shared with other classifiers according to the self-distillation technique. Such a fractional shot image classification program inputs a new input image into an image classification model and outputs a classification result of the new input image.

Meanwhile, the memory 120 stores various types of data generated in the process of executing an operating system for driving the minority shot image classification apparatus 100 or a minority shot image classification program. In this case, the memory 120 collectively refers to a nonvolatile storage device that continuously maintains stored information even when power is not supplied, and a volatile storage device that requires power to maintain the stored information.

The processor 130 executes a program stored in the memory 120, but controls the entire process according to the execution of the fractional shot image classification program. Each operation performed by the processor 120 will be described in more detail later.

The processor 130 may include all types of devices capable of processing data. For example, it may refer to a data processing device embedded in hardware having a circuit that is physically structured to perform a function represented by a code or command included in a program. As an example of the data processing device built into the hardware as described above, a microprocessor, a central processing unit (CPU), a processor core, a multiprocessor, and an application-specific application-specific device (ASIC) Integrated circuit) and processing devices such as a field programmable gate array (FPGA) may be covered, but the scope of the present invention is not limited thereto.

Meanwhile, the minority shot image classification apparatus 100 may further include a database 140 and the like, which stores or provides data necessary for execution of the minority shot image classification program under the control of the processor 130. Such a database may be included as a separate component from the memory 110 or may be built in a partial area of the memory 110.

In addition, the minority shot image classification apparatus 100 may further include an input/output module 150 and the like. Various input/output interfaces for the operation of the fractional shot image classification apparatus 100 correspond to this.

2 is a flowchart illustrating a method for classifying a prime number shot image according to another embodiment of the present invention.

First, when an input image is input (S210), data for the input image is augmented according to a self-mixing technique that generates a new input image by replacing a part of the input image with another part of the corresponding input image before performing feature extraction for this. The processing is performed (S220).

This can be expressed as an equation as follows.

[Equation 1]

와 같은 처리를 수행하는 변환 함수이며, 이미지

의 패치

을 다른 부분의 패치

로 대체하는 것을 의미한다.At this time, T is

It is a conversion function that performs the same processing as an image

Patch of

The other part of the patch

Means to replace with.

3 is a diagram for describing a self-augmentation technique used in a method for classifying a minority shot image according to an exemplary embodiment of the present invention.

In the input image on the left, data augmentation processing is performed according to a self-mixing technique in which some patches 300 of the corresponding input image are replaced with patches of other parts of the corresponding input image.

As such data augmentation processing is performed, a model learning process is performed to consider detailed features of images that were not previously considered important in order to find the correct answer corresponding to the object, and consequently, the performance of the feature extractor can be improved. Through this, by preventing the problem of memorizing the learning data that deep learning has previously, learning is performed to extract features that take into account the unique patterns of the input image.Through this method, it is meaningful even in images of categories that were not seen in the learning stage. You can expect the effect of extracting the pattern.

In addition to such data enhancement processing, a preprocessing process of automatically converting the size or shape of the image to the input image may be additionally performed.

Referring back to FIG. 2, the first step of generating a classification model for classifying the input images through the process of performing feature extraction on input images for which data augmentation has been performed and learning the extracted features based on deep learning learning. The learning step is performed (S230).

4 is a diagram illustrating a feature extraction process used in a method for classifying a prime number shot image according to an exemplary embodiment of the present invention.

As shown in FIG. 4, a classification model may be generated through a multi-feature extraction classifier having a plurality of sub-layers. In this case, the classifier uses a classification technique using cosine similarity. The use of a cosine classifier is advantageous in a fractional shot learning method that requires classifying a new category because the feature extractor induces learning toward lowering the change in distribution between data within a class.

In particular, in the present invention, since the feature is extracted using data that has undergone data augmentation processing according to self-mixing and a classification model is generated, as shown, several sub-layers (3-1, 4-1, 4-2) A classification model that additionally includes) can be created.

Referring back to FIG. 2, a second learning step is performed in which a new auxiliary classifier is created from the main classifier for inclusion in the classification model, and the independently output results of each classifier are shared with other classifiers according to the self-distillation technique. Thus, a final image classification model is generated (S240).

5 is a diagram illustrating a self-distillation technique used in a method for classifying a fractional shot image according to an exemplary embodiment of the present invention.

The self-distillation technique creates a sub-model with another auxiliary classifier from the middle layer of the deep learning model, and knows the prediction result output by this auxiliary classifier and the result predicted by the main classifier through KL divergence. It refers to the technique of sharing. For the input sample, each classifier outputs an independent classification result, and by sharing this information with each other, the main classifier is normalized to have a richer output distribution of information.

The related formula is as follows.

[Equation 2]

는 학습 데이터의 카테고리를,

분류기의 개수를,

는 학습 파라미터를 의미한다. 이와 같이, 자가증류 기법에서는 각 영상의 특징을 나타내는 특징 벡터를 분류하여 지식을 공유하도록 하고, 각 분류기에서 출력된 지식 정보인 각 클래스의 확률 분포가 비슷해지도록 정규화하여, 메인 분류 모델이 좀더 정확한 분류를 수행할 수 있도록 한다.In Equation 2

Is the category of training data,

The number of classifiers,

Denotes the learning parameter. In this way, in the self-distillation technique, feature vectors representing features of each image are classified to share knowledge, and by normalizing the probability distribution of each class, which is the knowledge information output from each classifier, to be similar, the main classification model is classified more accurately. To be able to perform.

Referring back to FIG. 2, a classification result is output for a newly input image using the image classification model constructed through the previous step (S250). An additional step of evaluating the performance of the image classification model may be performed using the classification result.

In such an evaluation step, features of the input image are extracted, but different from the first and second learning steps described above, the final feature is extracted using only a feature extractor of a single layer structure. In addition, for performance evaluation, a class to which the corresponding image belongs may be finally evaluated based on the similarity with the query image using a small number of input images that are not input in the learning stage.

In the present invention, the image classification model is generated through the first learning step and the second learning step as described above.

The deep learning-based fractional shot image classification method according to an embodiment of the present invention may be implemented in the form of a recording medium including instructions executable by a computer such as a program module executed by a computer. Computer-readable media can be any available media that can be accessed by a computer, and includes both volatile and nonvolatile media, removable and non-removable media. Further, the computer-readable medium may include a computer storage medium. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data.

Although the methods and systems of the present invention have been described in connection with specific embodiments, some or all of their components or operations may be implemented using a computer system having a general-purpose hardware architecture.

The foregoing description of the present application is for illustrative purposes only, and those of ordinary skill in the art to which the present application pertains will be able to understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present application. Therefore, it should be understood that the embodiments described above are illustrative and non-limiting in all respects. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as being distributed may also be implemented in a combined form.

The scope of the present application is indicated by the claims to be described later rather than the detailed description, and all changes or modified forms derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present application.

100: fractional shot image classification device
110: communication module
120: memory
130: processor
140: database
150: I/O module

Claims (7)

In the deep learning-based fractional shot image classification method,
Perform data augmentation processing on the input image according to a self-mixing technique that generates a new input image by replacing a part of the input image with another part of the input image, and performs feature extraction on the input images for which data augmentation has been performed. , A first learning step of generating a classification model for learning the extracted features and classifying input images based on deep learning learning;
A secondary learning step of generating a new auxiliary classifier from the main classifier for inclusion in the classification model, and allowing the independently output results of each classifier to be shared with other classifiers according to the self-distillation technique, and
And outputting a classification result by inputting a new input image to an image classification model that has undergone the first and second learning steps. The method of claim 1,
In the first learning step, the classification model is generated through a multi-feature extraction classifier having a plurality of sub-layers, and the classification model performs classification using cosine similarity. The method of claim 1,
In the secondary learning step, knowledge sharing is performed on a prediction result output by the auxiliary classifier and a result predicted by the main classifier through KL divergence. In the deep learning-based fractional shot image classification device,
Communication module;
A memory in which a fractional shot image classification program is stored;
Including a processor for executing a program stored in the memory,
The fractional shot image classification program performs data augmentation processing on the input image according to a self-mixing technique that generates a new input image by replacing a part of the input image with another part of the corresponding input image, and input images with data augmentation. A new auxiliary classifier from the main classifier for inclusion in the classification model and the primary learning step of generating a classification model for learning the extracted features based on deep learning learning and classifying the input image. And an image classification model generated through a secondary learning step in which the independently output results of each classifier are shared with other classifiers according to a self-distillation technique, and the fractional shot image classification program is a new input image Deep learning-based minority shot image classification apparatus for inputting the image classification model to output a classification result of a new input image. The method of claim 4,
In the first learning step, the classification model is generated through a multi-feature extraction classifier having a plurality of sub-layers, and the classification model performs classification using a cosine similarity. The method of claim 4,
In the secondary learning step, knowledge sharing is performed on a prediction result output from the auxiliary classifier and a result predicted by the main classifier through KL divergence. A non-transitory computer-readable recording medium in which a computer program for performing the deep learning-based fractional shot image classification method according to any one of claims 1 to 3 is recorded.

Images