KR102291111B1 - Zero Shot Recognition Apparatus Based on Self-Supervision and Method Thereof - Google Patents

Abstract

Zero-shot recognition capable of improving zero-shot recognition performance for a complex category data set by transforming attribute information on a single category and generating general-purpose feature information through an adversarial generation neural network based on the transformed attribute information Methods and apparatus are provided.

Description

Zero Shot Recognition Apparatus Based on Self-Supervision and Method Thereof}

The technical field to which this embodiment belongs relates to a zero-shot recognition apparatus and method.

The content described in this section merely provides background information for the present embodiment and does not constitute the prior art.

Zero-shot learning is a technology that can recognize unseen classes that are not included in the training data. The principle of the zero-shot learning technology is that if the model is trained to correctly infer the attribute information related to the class from the data of the class used for learning (Seen), the attribute information of the corresponding class even for the data of the first class By inferring the class, it is possible to recognize the class it sees for the first time.

The given attribute defined in the data set means the main characteristics representing the class, such as the length of the bird's beak, wing color, and body size. Existing zero-shot learning techniques use attribute information as key clues, so if the data set is not predefined for training, it affects the accuracy.

User-defined attribute information is one of the essential elements in classifying different categories in zero-shot learning. Newly defining attribute information for various categories requires the user's labor and professional knowledge, and requires a lot of time and money, so it is practically impossible for the user to set all of the attribute information.

The existing zero-shot learning model has a problem in that it cannot understand concepts that are not defined by the user at all. For example, when attribute information of “stripes” necessary for distinguishing between horses and zebras is not defined by the user, the existing zero-shot learning model cannot distinguish between horses and zebras.

A single-category data set is a data set that consists only of classes limited to one category (a set of classes). For example, a data set containing only flower types or a data set containing only bird types.

A complex category data set consists of two or more categories, and it is a data set in which it is difficult to define unified attribute information. It consists of different categories, including, for example, dogs, flowers, cars, furniture, etc.

A model trained in one category has limitations that cannot be used in other categories. Since the system of attribute information is different, a zero-shot model trained as a new type cannot be used for a data set of a flower type. There is no zero-shot learning model that can recognize other categories.

US 2018/0197050 (2018.07.12)

Embodiments of the present invention transform attribute information about a single category and generate general-purpose feature information through an adversarial generation neural network based on the transformed attribute information, to improve zero-shot recognition performance for a complex category data set. There is this.

Other objects not specified in the present invention may be additionally considered within the scope that can be easily inferred from the following detailed description and effects thereof.

According to one aspect of this embodiment, in the zero-shot recognition method by a computing device, converting original attribute information for a single category into new attribute information through a self-map-based attribute transformation model, using the new attribute information generating general-purpose feature information applicable to a complex category based on a loss function defined in the feature generation model, and transmitting the general-purpose feature information to the complex category-based zero-shot learning model. provide a way

According to another aspect of this embodiment, in the zero-shot recognition apparatus including one or more processors and a memory for storing one or more programs executed by the one or more processors, the processor self-maps raw attribute information for a single category converted into new attribute information through an attribute transformation model based on the attribute, and the processor generates general-purpose characteristic information applicable to a complex category based on a loss function defined in the characteristic creation model using the new attribute information, the processor comprising: There is provided a zero-shot recognition apparatus characterized in that the general-purpose characteristic information is transmitted to the zero-shot learning model based on the complex category.

As described above, according to the embodiments of the present invention, by transforming attribute information on a single category and generating general-purpose feature information through an adversarial generation neural network based on the transformed attribute information, zero-shot for a complex category data set It has the effect of improving recognition performance.

Even if it is an effect not explicitly mentioned herein, the effects described in the following specification expected by the technical features of the present invention and their potential effects are treated as if they were described in the specification of the present invention.

1 is a diagram illustrating an existing zero-shot learning model.
2 is a diagram illustrating a single category and a composite category.
3 is a block diagram illustrating a zero-shot recognition apparatus according to an embodiment of the present invention.
4 is a diagram illustrating an attribute conversion model of a zero-shot recognition apparatus according to an embodiment of the present invention.
5 is a diagram illustrating a feature generation model of a zero-shot recognition apparatus according to an embodiment of the present invention.
6 is a flowchart illustrating a zero-shot recognition method according to another embodiment of the present invention.
7 and 8 are diagrams illustrating simulation results according to embodiments of the present invention.

Hereinafter, in the description of the present invention, if it is determined that the subject matter of the present invention may be unnecessarily obscured as it is obvious to those skilled in the art with respect to related known functions, the detailed description thereof will be omitted, and some embodiments of the present invention will be described. It will be described in detail with reference to exemplary drawings.

1 is a diagram illustrating an existing zero-shot learning model.

Zero-shot learning, developed to solve the problem of not being able to recognize new labels that have not been trained with training data by existing deep learning, can recognize new data using other types of data, but the existing zero-shot learning is also If the attribute information is not defined, there is a problem that the new class cannot be recognized.

Existing zero-shot learning can only be learned from a data set in which attribute information is defined. A model trained in one category has limitations that cannot be used in other categories. For example, a zero-shot model trained on bird species cannot be used on a data set of flower types. This is because the defined attribute information system is different and there is no zero-shot learning technology that can recognize different categories.

Referring to FIG. 1 illustrating the existing zero-shot learning model, the conventional zero-shot learning trains the model to correctly infer attribute information related to the class from the data of the see class used for learning. Then, even for the data of the first class (Unseen Class), the property information of the class is inferred and the class seen for the first time is recognized.

The class used for learning (Seen Class) is a learning class used to learn property information, and the first class (Unseen Class) is a zero-shot class for testing that is not used at all in learning. A single-category data set is a data set that consists only of classes limited to one category (a set of classes). For example, a data set containing only flower types, a data set containing only bird types, etc.

The given attribute defined in the data set means the main attribute representing the class. For example, a bird's beak length, wing color, and body size. Or the length of the horse's legs, the shape of the head, the shape of the tail, etc. The zero-shot learning model uses attribute information as a key clue. If the data set is predefined for training, there is a problem in that the accuracy is lowered during zero-shot training.

Attribute information not defined in the data set (Not Given Attribute) means attribute information not defined in the existing data set.

2 is a diagram illustrating a single category and a composite category.

Existing zero-shot learning techniques are studied on single-category data sets, and the classification accuracy of zero-shot techniques is not high in complex category data sets.

A single-category data set is a data set that consists only of classes limited to one category (a set of classes). For example, a data set containing only flower types or a data set containing only bird types.

A complex category data set consists of two or more categories, and it is a data set in which it is difficult to define unified attribute information. It consists of different categories, including, for example, dogs, flowers, cars, furniture, etc.

A model trained in one category has limitations that cannot be used in other categories. Since the system of attribute information is different, a zero-shot model trained as a new type cannot be used for a data set of a flower type. There is no zero-shot learning model that can recognize other categories.

The present embodiment overcomes the limitations of the existing zero-shot learning method by transforming attribute information based on self-direction and generating general-purpose characteristic information based on the transformed attribute information. The present embodiment may recognize objects included in the complex category data set or objects not used in learning through general-purpose feature information.

3 is a block diagram illustrating a zero-shot recognition apparatus according to an embodiment of the present invention.

The zero-shot recognition apparatus 110 includes at least one processor 120 , a computer-readable storage medium 130 , and a communication bus 170 .

The processor 120 may control to operate as the zero-shot recognition apparatus 110 . For example, the processor 120 may execute one or more programs stored in the computer-readable storage medium 130 . The one or more programs may include one or more computer-executable instructions, which, when executed by the processor 120 , configure the zero-shot recognition device 110 to perform operations according to the exemplary embodiment. can be

Computer-readable storage medium 130 is configured to store computer-executable instructions or program code, program data, and/or other suitable form of information. The program 140 stored in the computer-readable storage medium 130 includes a set of instructions executable by the processor 120 . In one embodiment, the computer-readable storage medium 130 includes memory (volatile memory, such as random access memory, non-volatile memory, or a suitable combination thereof), one or more magnetic disk storage devices, optical disk storage devices, It may be flash memory devices, other types of storage media that can be accessed by the zero-shot recognition apparatus 110 and store desired information, or a suitable combination thereof.

The communication bus 170 interconnects various other components of the zero-shot recognition device 110 , including the processor 120 and the computer-readable storage medium 140 .

The zero-shot recognition device 110 may also include one or more input/output interfaces 150 and one or more communication interfaces 160 that provide interfaces for one or more input/output devices 24 . The input/output interface 150 and the communication interface 160 are connected to the communication bus 170 . The input/output device (not shown) may be connected to other components of the zero-shot recognition device 110 through the input/output interface 150 .

The zero-shot recognition apparatus 110 converts original attribute information for a single category into new attribute information through a self-map-based attribute transformation model, and uses the new attribute information to create a complex based on a loss function defined in the feature generation model. Generating general-purpose characteristic information applicable to categories, transmitting general-purpose characteristic information to a composite category-based zero-shot learning model, and recognizing unused class data based on general-purpose characteristic information through the zero-shot learning model.

4 is a diagram illustrating an attribute conversion model of a zero-shot recognition apparatus according to an embodiment of the present invention.

The attribute transformation model is a model that transforms input attribute information at a vector level. A plurality of samples are input at a vector level to constitute an embedding space.

Referring to FIG. 4A , original attribute information is input as a plurality of vectors. The attribute transformation model transforms some values into other values in multiple vectors.

Referring to FIG. 4B , the attribute transformation model may convert some values of a plurality of vectors to zero (0).

Referring to FIG. 4C , the attribute conversion model may convert some values of a plurality of vectors into noise values. Gaussian noise may be applied as the noise value.

Referring to FIG. 4D , the attribute transformation model mutually substitutes two corresponding values in a plurality of vectors. Other attribute information values corresponding to the same column are replaced in units of columns. Unlike zero or noise, real values are used, so a manifold of raw attribute information can be maintained. In the field of learning, a manifold is a space with data, and it means a connection or group of points expressed from high to low dimensions.

5 is a diagram illustrating a feature generation model of a zero-shot recognition apparatus according to an embodiment of the present invention.

For general-purpose zero-shot learning that is not limited to human-defined attribute information-based data sets, a feature generation model is trained using original attribute information and changed attribute information.

The feature generation model receives random noise distribution and new attribute information and outputs general-purpose feature information. In an adversarial generative neural network where the generative model and the discriminant model interact, (i) a conditionally generated loss function and (ii) a self-guided loss function By optimizing, the general-purpose characteristic information is corrected.

The conditionally generated loss function (L _gan ) is expressed as in Equation (1), and the self-guided loss function (L _ss ) is expressed as in Equation (2).

The conditionally generated loss function applies a Wasserstein distance to the general-purpose feature information conditionally with a penalty weight.

는 생성 모델의 분포 P_g로부터 클래스 y의 생성된 특징이다. Z는 특징 생성 모델의 자유도를 높이는 노이즈 벡터이다.

는

조건에 따른 처음 보는 클래스의 CNN 특징 정보이다. λ는 페널티 가중치 파라미터이다. 로그 우도(Log Likelihood) 대신에 바서스타인 거리를 적용하고, 경사 페널티를 추가로 포함한다. 바서스타인 거리는 두 확률분포의 연관성을 측정하여 그 거리의 기대값이 가장 작을 때의 거리를 의미한다. 결합 확률분포는 두 분포가 동시에 일어날 때의 사건에 대한 확률분포를 의미한다. x means CNN feature information of class y.

is the generated feature of class y from _{the distribution P g} of the generative model. Z is a noise vector that increases the degree of freedom of the feature generation model.

Is

This is the CNN feature information of the first class you see according to the condition. λ is a penalty weight parameter. Instead of log likelihood (Log Likelihood), we apply the Basherstein distance, and additionally include the slope penalty. The Wasserstein distance measures the correlation between two probability distributions and means the distance when the expected value of the distance is the smallest. A joint probability distribution refers to a probability distribution for an event when two distributions occur simultaneously.

)를 적용한다. 속성 정보에 대한 치환 유무를 분별하는 목적 함수이다. 속성 변환 모델은 특정 확률 상수 p에 의해 열 단위로 속성 값을 섞는 함수이며, 매니폴드를 그대로 유지하면서 이상치(outlier) 또는 노이즈에 강인하다.The self-guidance loss function is the distribution (

) is applied. This is an objective function that distinguishes whether or not to replace attribute information. The attribute transformation model is a function that mixes attribute values in units of columns by a specific probability constant p, and is robust against outliers or noise while maintaining the manifold as it is.

All losses are applicable to complex categorical data sets, resulting in the creation of features that are robust against outliers and noise.

When the processor transmits pseudo attribute information to the zero-shot learning model, the zero-shot learning model recognizes unused class data based on general-purpose feature information.

We use the user-defined attribute space as the embedding space and the compatibility score s(y) for class y.

W is a weight matrix having a fully connected layer. In order to predict the class level for a given feature x, the zero-shot learning model can be projected onto the attribute expression. The compatibility score s(y) can be used to select the ^{best matched class y*.}

조건에서, 가장 높은 양립 가능성 점수를 가는 y^*는 예측된 클래스이다.

로 ZSL(Zero Shot Learning)을 설정할 수 있고,

로 GZSL(Generalized Zero Shot Learning)를 설정할 수 있다. GZSL는 테스트 단계에서 처음 보는 클래스(

, Unseen Class)와 사용된 클래스(

, Seen Class)를 동시에 분류한다.

In the condition, y ^* with the highest compatibility score is the predicted class.

You can set ZSL (Zero Shot Learning) with

You can set GZSL (Generalized Zero Shot Learning). GZSL is the first class (

, Unseen Class) and the class used (

, Seen Class) at the same time.

The zero-shot learning model may include a feature extraction model, and the feature extraction model may be implemented as a convolutional neural network (CNN). In the feature extraction model, multiple layers are networked and include hidden layers. A layer may include a parameter, and the parameter of the layer includes a set of learnable filters. A convolution filter can be applied as a filter. The parameters include weights and/or biases between nodes.

6 is a flowchart illustrating a zero-shot recognition method according to another embodiment of the present invention. The zero-shot recognition method may be performed by a computing device, and operates in the same manner as the zero-shot recognition apparatus.

In step S210, the processor converts original attribute information for a single category into new attribute information through a self-map-based attribute transformation model. The original attribute information is input as a plurality of vectors. The attribute transformation model transforms some values into other values in multiple vectors. The attribute transformation model may transform some values in a plurality of vectors into zero or noise values. The attribute transformation model may maintain a manifold of original attribute information by mutually permuting two corresponding values in a plurality of vectors.

In step S220, the processor generates general-purpose feature information applicable to a complex category based on the loss function defined in the feature generation model by using the new attribute information. The feature generation model receives the random noise distribution and the new attribute information and outputs the general feature information. The general-purpose feature information is corrected by optimizing the function. The conditionally generated loss function may apply a Wasserstein distance to general-purpose feature information conditionally with a penalty weight. The self-guidance loss function may apply the distribution of the discriminant model to the general-purpose feature information according to the original attribute information or the new attribute information while performing the self-guidance.

In step S220, the processor transmits general-purpose feature information to the composite category-based zero-shot learning model.

In step S220, when the processor transmits pseudo attribute information to the zero-shot learning model, the zero-shot learning model may recognize data of an unused class based on general-purpose feature information.

7 and 8 are diagrams illustrating simulation results according to embodiments of the present invention.

For the quantitative evaluation of the automatically generated data and attribute information of the first-time class, the Average Per-Class Top-1 Accuracy (%) measurement method widely used in zero-shot learning was used. Average Per-Class Top-1 Accuracy (%) calculates the average value of recognition accuracy (%) for each class and uses it as a measure of final zero-shot performance evaluation.

ImageNet 21K, a composite category zero-shot data set, was used, and a total of 9 sets of test classes were used for evaluation. 2H is the set of all classes within 2 spaces from the learning class in the WordNet hierarchy among all 21K classes. 3H is the set of all classes within 3 spaces from the learning class in the WordNet hierarchy among all 21K classes. M500 is a set of 500 classes with the largest amount of data among all 21K classes. M1K is a set of 1,000 classes with the largest amount of data among all 21K classes. M5K is a set of 5,000 classes with the largest amount of data among all 21K classes. L500 is a set of 500 classes with the smallest amount of data among all 21K classes. L1K is a set of 1,000 classes with the smallest amount of data among all 21K classes. L5K is a set of 5,000 classes with the smallest amount of data among all 21K classes. All20K uses all 21K classes.

The zero-shot recognition apparatus according to this embodiment shows the best result of Top-1 classification accuracy of the latest zero-shot technology for ImageNet, which is a kind of complex category data set. The zero-shot recognition apparatus according to the present embodiment does not limit the type of category and the range of characteristic information, and can overcome the limitations of existing technologies.

The zero-shot recognition apparatus may be implemented in a logic circuit by hardware, firmware, software, or a combination thereof, or may be implemented using a general-purpose or special-purpose computer. The device may be implemented using a hardwired device, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), or the like. In addition, the device may be implemented as a system on chip (SoC) including one or more processors and controllers.

The zero-shot recognition apparatus may be mounted in the form of software, hardware, or a combination thereof on a computing device or server provided with hardware elements. A computing device or server is all or part of a communication device such as a communication modem for performing communication with various devices or a wired/wireless communication network, a memory for storing data for executing a program, and a microprocessor for executing operations and commands by executing the program It can mean a variety of devices, including

In Figures 5 and 6, each process is described as being sequentially executed, but this is merely an exemplary description, and those skilled in the art are shown in Figures 5 and 6 within the range that does not depart from the essential characteristics of the embodiment of the present invention. Various modifications and variations may be applied by changing the order described, executing one or more processes in parallel, or adding other processes.

The operations according to the present embodiments may be implemented in the form of program instructions that can be performed through various computer means and recorded in a computer-readable medium. Computer-readable media refers to any medium that participates in providing instructions to a processor for execution. Computer-readable media may include program instructions, data files, data structures, or a combination thereof. For example, there may be a magnetic medium, an optical recording medium, a memory, and the like. A computer program may be distributed over a networked computer system so that computer readable code is stored and executed in a distributed manner. Functional programs, codes, and code segments for implementing the present embodiment may be easily inferred by programmers in the technical field to which the present embodiment pertains.

The present embodiments are for explaining the technical idea of the present embodiment, and the scope of the technical idea of the present embodiment is not limited by these embodiments. The protection scope of this embodiment should be interpreted by the following claims, and all technical ideas within the equivalent range should be interpreted as being included in the scope of the present embodiment.

Claims (14)

A zero-shot recognition method by a computing device, comprising:
converting original attribute information for a single category into new attribute information through a self-map-based attribute transformation model;
generating general-purpose feature information applicable to a complex category based on a loss function defined in a feature creation model using the new attribute information; and
Transmitting the general-purpose feature information to the composite category-based zero-shot learning model,
The feature generation model receives a random noise distribution and the new attribute information, outputs the general-purpose feature information, and conditionally applies (i) a penalty weight defined in an adversarial generative neural network in which the generative model and the discriminant model interact. A conditionally generated loss function that applies Wasserstein Distance to information and (ii) the distribution of the discriminant model for the universal feature information according to the original attribute information or the new attribute information while performing self-map A zero-shot recognition method, characterized in that the general-purpose feature information is corrected using a result of learning a self-guided loss function. According to claim 1,
The original attribute information is input as a plurality of vectors,
and the attribute transformation model converts some values in the plurality of vectors into other values. 3. The method of claim 2,
and the attribute conversion model converts some values of the plurality of vectors into zero or noise values. 3. The method of claim 2,
and the attribute transformation model maintains a manifold of the original attribute information by mutually permuting two values corresponding to the plurality of vectors. delete delete delete According to claim 1,
When pseudo-attribute information is sent to the zero-shot learning model,
The zero-shot learning model recognizes data of an unused class based on the general-purpose feature information. A zero-shot recognition device comprising one or more processors and a memory for storing one or more programs executed by the one or more processors,
The processor converts original attribute information for a single category into new attribute information through a self-map-based attribute transformation model,
The processor generates general-purpose feature information applicable to a complex category based on a loss function defined in a feature creation model using the new attribute information,
The processor transmits the general-purpose feature information to the composite category-based zero-shot learning model,
The feature generation model receives a random noise distribution and the new attribute information, outputs the general-purpose feature information, and conditionally applies (i) a penalty weight defined in an adversarial generative neural network in which the generative model and the discriminant model interact. A conditionally generated loss function that applies Wasserstein Distance to information and (ii) the distribution of the discriminant model for the universal feature information according to the original attribute information or the new attribute information while performing self-map A zero-shot recognition apparatus, characterized in that the general-purpose feature information is corrected using a result of learning the self-guided loss function. 10. The method of claim 9,
The original attribute information is input as a plurality of vectors,
and the attribute transformation model maintains a manifold of the original attribute information by mutually permuting two values corresponding to the plurality of vectors. delete delete delete 10. The method of claim 9,
When the processor transmits pseudo attribute information to the zero-shot learning model,
The zero-shot learning model recognizes data of an unused class based on the general-purpose feature information.

Images