KR102458783B1 - Generalized zero-shot object recognition device and generalized zero-shot object recognizing method - Google Patents

Abstract

A generalized zero-shot object recognition apparatus according to an embodiment of the present invention includes an object learner that learns input first image data using hierarchical knowledge of classes stored in a dataset, and second image data that is input to the object. and an object estimator that adjusts a possibility of being classified into an unlearned class by a learner, wherein the object estimator calculates an uncertainty estimate indicating a probability that input image data has been previously learned separately, and the input image The possibility of being classified into an unlearned class by the object learner is adjusted using the uncertainty estimation value.

Description

Generalized zero-shot object recognition device and generalized zero-shot object recognition method

The present invention relates to a supervised learning-based generalized zero-shot object recognition apparatus and object recognition method, and more specifically, to improve zero-shot object recognition performance, and to prevent performance degradation of unused class data during learning. The present invention relates to a generalized zero-shot object recognition apparatus and an object recognition method.

Recently, in the field of machine learning, various learning methodologies focusing on deep learning models have been proposed, and many of them use supervised learning methodologies based on educational data with class labels.

Among them, the zero-shot learning methodology is a methodology that can perform recognition on a class that has not been learned in the learning stage of supervised learning. A properly trained zero-shot learning model can properly recognize classes that are not included in the training data (unseen classes) in the inference stage, but if we focus only on processing the data of these classes (unseen classes), Performance of processing data of used classes (seen classes) may be degraded.

Therefore, a method of applying generalized zero-shot learning considering both data of classes used for learning and classes not used for learning is proposed.

There are two main approaches to zero-shot training for object recognition models.

The first approach for zero-shot learning is to use image data along with attributes for that image for training. However, this approach has a disadvantage in that in the process of building training data, labels and attributes for image data must be built.

The second approach for zero-shot learning is to embed images and class labels for images in a common vector space, and it is a method to perform meaningful embeddings by appropriately utilizing a kind of text knowledge for class labels. have. However, this approach has the disadvantage of showing relatively low performance compared to the approach using image properties because zero-shot classification must be performed only depending on the word embedding result in the language model.

In addition, all of the major approaches introduced above are generalized zero-shot classification, which performs recognition with both see classes and unseen classes used during training as classes. When performing , there is a problem that a kind of over-fitting of the classes used during training occurs, and the classification performance of classes not used during training is relatively significantly reduced.

SUMMARY OF THE INVENTION It is an object of the present invention to improve the recognition performance of an object recognition model for an object to be seen for the first time. In addition, it is to minimize the degradation of generalized zero-shot object recognition performance for unused class data during training.

A generalized zero-shot object recognition apparatus according to an embodiment of the present invention includes an object learner that learns input first image data using hierarchical knowledge of classes stored in a dataset, and second image data that is input to the object. and an object estimator that adjusts a possibility of being classified into an unlearned class by a learner, wherein the object estimator calculates an uncertainty estimate indicating a probability that input image data has been previously learned separately, and the input image The possibility of being classified into an unlearned class by the object learner is adjusted using the uncertainty estimation value.

The object learner, a first image embedding unit including a neural network layer that converts the first image data into a first image vector of a specific dimension, considers the classes stored in the dataset as text, and converts the text into a first image vector of a specific dimension A first text embedding unit comprising a neural network layer for converting one text vector into a hierarchical semantic vector, receiving text information of the classes from the first text embedding unit, and writing the text information A hierarchical knowledge generating unit generating the hierarchical knowledge compared with the stored hierarchical structure and transmitting it to the first text embedding unit, and receiving the first image vector, the first text vector, and the hierarchical semantic vector, each It includes a calculator for calculating the Euclidean distance.

The hierarchical knowledge includes upper class information and sibling class information of the classes, and the object learner learns so that the first image vector and the first text vector are close to a hierarchical semantic vector of the upper class. do.

The hierarchical knowledge includes superclass information and sibling class information of the classes, and the object learner learns so that the first image vector and the first text vector are far apart from the hierarchical semantic vector of the sibling class. do.

The first image vector and the first text vector have the same dimension.

The object estimator includes a neural network layer for classifying the second image data as separate training data, an uncertainty estimator for calculating the uncertainty estimate value from the distribution of the classified result value, and the second image data A second image embedding unit including a neural network layer for converting a second image vector of a specific dimension, the classes learned by the object learner are regarded as text and converted into a second text vector of a specific dimension, by the object learner a neural network layer that considers unlearned classes as text and converts them into a third text vector of a specific dimension, and a learning distance defined by a text embedding unit and a distance value between the second image vector and the second text vector; and a distance comparator dividing an unlearned distance defined as a distance between the third text vector and the second image vector by the uncertainty estimate value.

The uncertainty estimation value has a lower value as the likelihood that the second image data is included in the training data increases, and has a higher value as the likelihood that the second image data is included in the training data is low.

The second image vector, the second text vector, and the third text vector have the same dimension.

The learning data is learned with the same data as the class data used by the object learner.

A generalized zero-shot object recognition method according to an embodiment of the present invention includes the steps of converting first image data into a first image vector in a first image embedding unit and transmitting it to an operation unit, at least stored in a dataset by the first text embedding unit Transmitting text information of some classes to a hierarchical knowledge generating unit, transmitting, by the hierarchical knowledge generating unit, hierarchical knowledge of the class text information to the first text embedding unit, wherein the first text embedding unit regards the classes as text converting the text into a first text vector of a specific dimension and transmitting it to the operation unit, the first text embedding unit converting the hierarchical knowledge into a hierarchical semantic vector and transmitting it to the operation unit, the operation unit first calculating a hierarchical semantic loss value by calculating each Euclidean distance from the image vector, the first text vector, and the hierarchical semantic vector, and learning parameters such that the hierarchical semantic loss value is minimized. include

The transmitting of the hierarchical knowledge to the first text embedding unit includes: setting a hierarchical structure and storing the hierarchical knowledge generating unit; and generating hierarchical knowledge, wherein the hierarchical knowledge includes superclass information and sibling class information of the classes.

The hierarchical semantic loss value decreases as the distance between the first image vector and the first text vector is closer to the hierarchical semantic vector of the higher class.

The hierarchical semantic loss value decreases as the distance between the first image vector and the first text vector increases from the hierarchical semantic vector of the sibling class.

The generalized zero-shot object recognition method according to an embodiment of the present invention classifies the second image data as separate learning data in the uncertainty estimator to determine the probability values that the second image data is classified into each of the classes included in the learning data. outputting, the uncertainty estimation unit calculating an uncertainty estimation value from the distribution of the probability values and transmitting it to the distance comparator, converting the second image data into a second image vector in a second image embedding unit and transmitting the second image data to the calculation unit , converting the learned classes into a second text vector by a second text embedding unit considering all classes stored in the dataset as text, and converting unlearned classes into a third text vector, the second image vector Calculating a learning distance defined as a distance value between and the second text vector and an unlearned distance defined as a distance between the third text vector and the second image vector, and transmitting it to the distance comparator; and The method further includes calculating, by a distance comparison unit, a final classification result by dividing the unlearned distance by the uncertainty estimate.

The uncertainty estimation value has a lower value as the likelihood that the second image data is included in the training data increases, and has a higher value as the likelihood that the second image data is included in the training data is low.

The second image vector, the second text vector, and the third text vector have the same dimension.

According to the present invention, recognition performance of an object to be seen for the first time may be improved. The zero-shot performance may be increased through the object learner, and the generalized zero-shot performance may be increased through the object estimator. In addition, the application efficiency of the deep learning-based object recognition model can be improved.

1 is a schematic diagram of an object recognition apparatus according to an embodiment of the present invention.
FIG. 2 is a diagram schematically illustrating the object learner shown in FIG. 1 .
3 is a diagram schematically illustrating a learning method of an object learner according to an embodiment of the present invention.
4 is a graph of measuring classification accuracy for images of classes not used during learning.
FIG. 5 is a diagram schematically illustrating the object estimator shown in FIG. 1 .
6 is a diagram illustrating an object recognition method according to an embodiment of the present invention.

All embodiments described below are illustratively shown to aid understanding of the present invention, and may be modified differently from the embodiments described herein and implemented in various embodiments. In addition, in describing the present invention, if it is determined that a detailed description of a related known function or known component may unnecessarily obscure the gist of the present invention, the detailed description thereof will be omitted.

The accompanying drawings are not drawn to scale in order to help the understanding of the invention, but the dimensions of some components may be exaggerated. Even though they are indicated in , they are indicated with the same symbols as possible.

In addition, in describing the components of the embodiment of the present invention, terms such as first, second, A, B, (a), (b), etc. may be used. These terms are only for distinguishing the elements from other elements, and the essence, order, or order of the elements are not limited by the terms. When it is described that a component is 'connected', 'coupled' or 'connected' to another component, the component may be directly connected, coupled or connected to the other component, but the component and the other component It should be understood that another element may be 'connected', 'coupled' or 'connected' between elements.

Therefore, since the embodiments described in this specification and the configurations shown in the drawings are only the most preferred embodiments of the present invention and do not represent all the technical spirit of the present invention, there may be various modified embodiments of the present invention. .

And, the terms or words used in the present specification and claims should not be limited to conventional or dictionary meanings, and the inventor may properly define the concept of the term to describe his invention in the best way. Based on the principle, it should be interpreted as meaning and concept consistent with the technical idea of the present invention.

In addition, the present invention is not limited to the embodiments disclosed below, but will be implemented in a variety of different forms, only these embodiments allow the disclosure of the present invention to be complete, and common knowledge in the technical field to which the present invention belongs It is provided to fully inform the possessor of the scope of the invention, and the present invention is only defined by the scope of the claims.

Also, the singular expression used in this application includes the plural expression unless the context clearly dictates otherwise.

1 is a schematic diagram of an object recognition apparatus according to an embodiment of the present invention, and FIG. 2 is a diagram schematically illustrating the object learner shown in FIG. 1 .

1 and 2 , an object recognition apparatus 1000 according to an embodiment of the present invention may be a generalized zero-shot object recognition apparatus. That is, the object recognition apparatus 1000 according to an embodiment of the present invention applies a generalized zero-shot learning methodology, so that classes used for learning (seen classes) and classes not used for learning Objects can be recognized by considering all data of (unssen classes). For convenience of description, the generalized zero-shot object recognition apparatus 1000 is referred to as the object recognition apparatus 1000 in the specification to be described later.

The object recognition apparatus 1000 includes an object learner 100 and an object estimator 200 . The object learner 100 may learn so that input image data is more likely to be classified into a correct answer class among the classes stored in the dataset.

Specifically, the object learner 100 includes a first image embedding unit 110 , a first text embedding unit 120 , a hierarchical knowledge generation unit 130 , and an operation unit 140 .

The first image embedding unit 110 includes a neural network layer that converts first image data IM1 that is an input image into a first image vector x _s of a specific dimension. The input image may be unlearned data.

The first text embedding unit 120 considers at least some classes TX1 among the classes stored in the dataset as text, and converts the text TX1 into a first text vector (y _s ⁺ ) of a specific dimension. include layers.

According to this embodiment, the vector output dimension of the first image embedding unit 110 is set to be the same as the vector output dimension of the first text embedding unit 120 . That is, the first image vector (x _s ) and the first text vector (y _s ⁺ ) may be vectors of the same dimension. Accordingly, the distance between the first image vector (x _s ) and the first text vector (y _s ⁺ ) can be calculated in a common vector space.

The hierarchical knowledge generating unit 130 and the calculating unit 140 may be defined as a hierarchical loss function unit LF. The hierarchical loss function unit LF may use the hierarchical knowledge TK between classes to calculate a hierarchical semantic loss value including distance information between input vectors. The object learner 100 according to the present embodiment includes a first image embedding unit 110 and a first text embedding unit 120 such that the size of the hierarchical loss of meaning calculated through the hierarchical loss function unit LF is reduced. can learn the embedding model of

Specifically, the hierarchical knowledge generating unit 130 stores hierarchical structure information including class texts along with information on class texts of pre-learned data. The hierarchical structure information includes information of an upper class and sibling classes of the input class. Illustratively, when the information about the input class text is 'tiger', the upper class may be 'mammal' or 'predator', and the sibling class may be 'cheetah', 'cat', 'lion', and the like.

According to this embodiment, the hierarchical structure information may be arbitrarily designed in advance by the user. However, the present invention is not particularly limited to the method of constructing the hierarchical structure information. Illustratively, according to another embodiment of the present invention, the hierarchical structure information may be constructed by a designer when the learning data is initially designed, and is borrowed from another model (eg, WordNet) that generalizes the hierarchical structure of class texts. may be

The hierarchical knowledge generator 130 receives text information TI of the classes TX1 from the first text embedding unit 120 and compares the text information TI with a pre-stored hierarchical structure to obtain hierarchical knowledge (HK). create The hierarchical knowledge (HK) includes information about the upper class and the sibling class of the input class (TX1). The hierarchical knowledge HK is transmitted to the first text embedding unit 120 .

The first text embedding unit 120 converts the hierarchical knowledge HK received from the hierarchical knowledge generating unit 130 into a hierarchical semantic vector. Specifically, the first text embedding unit 120 provides an upper class vector (y _s ^sc ) defined as a hierarchical semantic vector of upper class information and a sibling class vector (y _s ^- ) defined as a hierarchical semantic vector of sibling class information. ) is created.

The operation unit 140 receives the first image vector (x _s ) from the first image embedding unit 110 , and receives the first text vector (y _s +) and the upper class vector (y) from the first text embedding unit 120 . _s ^sc ) and a sibling class vector (y _s ^- ). According to the present embodiment, the calculator 140 may calculate the Euclidean distance of each of the received vectors to calculate the hierarchical loss of meaning (HSL).

3 is a diagram schematically illustrating a learning method of an object learner according to an embodiment of the present invention.

Referring to FIG. 3 together with FIGS. 1 and 2 , the operation unit 140 calculates a hierarchical mean squared loss function HMSE (Hierarchical Mean Squared Loss) and a hierarchical triplet loss function HTL (Hierarchical Triplet Loss) respectively to calculate the total A hierarchical semantic loss (HSL), which is a loss value, may be calculated.

Equation 1 is a first image vector (x _s ) belonging to a class of training data, a first text vector (y _s ⁺ ) of a class corresponding to the first image vector (x _s ), and a first text vector (y _s ⁺ ) ), the Euclidean distance ^between the first image vector (x _s ) and the upper class vector (y _s ^sc ) and the first text vector (y _{s )} ⁺ ) and the hierarchical mean square error loss function (HMSE) that calculates the Euclidean distance between the upper class vector (y _s ^sc ).

를 입력으로 하여, 제1 이미지 벡터(x_s)에 대해서 제1 텍스트 벡터(y_s ⁺)를 양성 샘플로 설정하고, 형제 클래스 벡터(y_s ^-)를 음성 샘플로 설정하여 삼중항 손실을 계산하는 계층적 삼중항 손실함수(HTL)를 나타낸다.Equation 2 corresponds to the first image vector (x _s ), the first text vector (y _s ⁺ ) of the class corresponding to the first image vector (x _s ), and the sibling class of the first text vector (y _s ⁺ ) sibling class vector (y _s ^- ) and constant acting as margin

With as input, the triplet loss is calculated by setting the first text vector (y _s ⁺ ) as positive samples and the sibling class vector (y _s ^- ) as negative samples for the first image vector (x _s ). represents a hierarchical triplet loss function (HTL).

를 적용한 후 합한 계층적 의미 손실값 (HSL)을 나타낸다.Equation 3 is a constant reflecting the ratio of the hierarchical mean square error loss function (HMSE) and the hierarchical triplet loss function (HTL)

It represents the summed hierarchical loss of meaning (HSL) after applying .

According to this embodiment, the object learner 100 can learn the embedding model of the first image embedding unit 110 and the embedding model of the first text embedding unit 120 so that the loss value of the HSL with respect to the input data becomes small. have. That is, the object learner 100 uses the embedding model and the first text of the first image embedding unit 110 so that the hierarchical squared error loss function (HMSE) value decreases and the hierarchical triplet loss function (HTL) value increases. An embedding model of the embedding unit 120 may be learned. The hierarchical squared error loss function (HMSE) value decreases as the distance between the first image vector (x _s ) and the first text vector (y _s ⁺ ) is closer to the upper class vector (y _s ^sc ), and the hierarchical triplet The term loss function (HTL) value decreases as the distance increases.

The learning data TD on which learning is completed in the object learner 100 is transmitted to the object estimator 200 .

According to an embodiment of the present invention, the unlearned first image data IM1 is set as a recognition candidate, and the first image data IM1 input to the object learner 100 is classified using hierarchical knowledge, so zero Zero-shot object recognition performance may be improved.

4 is a graph of measuring classification accuracy for images of classes not used during learning. Specifically, the graph shown in FIG. 4 is "cat-dog", "plane-automobile", "automobile-deer", "deer-ship", "cat" in the CIFAR10 dataset consisting of color images for 10 classes. -truck" is set as unseen classes for each learning, and only the other 8 classes among the 10 classes are set as the classes used for learning and learning is performed, This is a comparison graph of the results of measuring classification accuracy for images of classes not used during training.

Referring to FIG. 4 , the object learner 100 according to an embodiment of the present invention has increased the classification accuracy of the embedding-based zero-shot object learner compared to the existing one, and between classes with high similarity to each other (eg, "cat- dog"), the difference in classification accuracy with the comparative example was remarkable. This can be interpreted as performance improvement through learning of the embedding model using the hierarchical structure of text vectors, unlike the existing methodologies that depend only on semantic knowledge of text vectors of language models.

FIG. 5 is a diagram schematically illustrating the object estimator shown in FIG. 1 .

Referring to FIG. 5 together with FIG. 1 , the data TD learned by the object learner 100 is transmitted to the object estimator 200 . The object estimator 200 may adjust the possibility that data input to the object estimator 200 is classified into a class that has not been learned by the object learner 100 .

Specifically, the object tracker 200 includes an uncertainty estimation unit 210 , a second image embedding unit 220 , a second text embedding unit 230 , and a distance comparison unit 240 .

The uncertainty estimator 210 includes a general neural network layer that classifies the second image data IM2, which is an input image, into separate learning data learned in advance using the object learner 100 .

The classified result value may be a set of values obtained by quantifying the similarities for each of the classes. This is defined as a connection weight. According to the present embodiment, the uncertainty of the image newly input to the object estimator 200 may be estimated from the distribution of the classified result values. A value for the uncertainty is defined as an uncertainty estimation value (S _uc ). The uncertainty estimation value S _uc may be transmitted to the distance comparator 240 in the form of a value calculated by applying a softmax function for the separate training data to the second image data IM2 .

When the distribution of the classified result values has a large value in a specific class, since classification is performed relatively reliably on the input image, the uncertainty estimator 210 can greatly calculate the possibility that the image is included in the training data. can On the other hand, when the distribution of the classified result values has uniform values in the entire class, the classification is performed relatively uncertainly with respect to the input image. can be calculated small.

)로 구성된 객체추정기(

)에 입력 이미지 (

)가 입력되었을 때, 객체추정기(

)의 출력값에 대해서 분포를 부드럽게 바꿔주는 상수 T를 적용한 소프트맥스 함수(softmax)를 적용하고 그 출력값들 중 가장 큰 값을 상수 2에서 뺀 불확실성 추정값(S_uc)를 계산하는 수식이다.Equation (4) is the connection weight (

) consisting of an object estimator (

) to the input image (

) is input, the object estimator (

), a softmax function to which a constant T that smoothly changes the distribution is applied is applied to the output value of ), and the uncertainty estimation value (S _uc ) is calculated by subtracting the largest value among the output values from the constant 2.

)가 입력데이터(

)를 보다 확실하게 분류할수록 1에 가까워지며, 반대로 불확실하게 분류할수록 2에 가까워지도록 설계된다. 따라서, 입력데이터가 학습 데이터의 클래스에 포함되지 않을 가능성이 커질수록 큰 값을 갖게 된다.The uncertainty estimation value (S _uc ) of Equation 4 is configured to equal a real value between 1 and 2, and the object estimator (

) is the input data (

) is designed to be closer to 1 as it is classified more reliably, and to be closer to 2 as it is classified with uncertainty. Therefore, as the probability that the input data is not included in the class of the learning data increases, it has a larger value.

The second image embedding unit 220 includes a neural network layer that converts second image data IM2 that is an input image into a second image vector x _q of a specific dimension. The second image data IM2 may include unlearned data and may be different from the first image data IM1 . In an embodiment of the present invention, the second image embedding unit 220 may be the same as the above-described first image embedding unit 110 .

The second text embedding unit 230 considers the classes TX2-1 learned by the object learner 100 among the classes stored in the dataset as text and converts them into a second text vector y _s of a specific dimension. and a neural network layer that considers the classes (TX2-2) not learned by the object learner 100 among the classes stored in the dataset as text and converts them into a third text vector (y _u ) of a specific dimension. . In an embodiment of the present invention, the second text embedding unit 230 may be the same as the above-described first text embedding unit 120 .

According to this embodiment, the vector output dimension of the second image embedding unit 220 is set to be the same as the vector output dimension of the second text embedding unit 230 . That is, the second image vector (x _q ), the second text vector (y _s ), and the third text vector (y _u ) may be vectors of the same dimension. Accordingly, the distance between the second image vector (x _q ), the second text vector (y _s ), and the third text vector (y _u ) can be calculated in a common vector space.

The distance comparison unit 240 receives the second image vector (x _q ) from the second image embedding unit 220 , and the second text vector (y _s ) and the third text vector from the second text embedding unit 230 . (y _u ) is received. According to this embodiment, the distance comparator 240 may calculate a value obtained by calculating the Euclidean distance of each of the received vectors. Specifically, the distance comparison unit 240 is a learning distance (D _s ) defined as a distance value between the second image vector (x _q ) and the second text vector (y _s ) and the second image vector (x _q ) and An unlearned distance D _u defined as a distance value between the third text vectors y _u is calculated. In this case, the learning distance D _s and the non-learning distance D _u may be temporary classification results of the object estimator 200 .

According to an embodiment of the present invention, the distance comparator 240 divides the unlearned distance D _u by the uncertainty estimate value S _uc received from the uncertainty estimator 210 , and the image input to the object estimator 200 . The possibility that the data IM2 is classified into a class not used during training may increase. That is, by reducing the distance value between the third text vector (y _u ) and the second image vector (x _q ), which are text vectors of the classes (TX2-2) not learned by the object learner 100, the input It is possible to increase the possibility that the image IM2 is classified into the unlearned classes TX2-2.

간의 유클리디언 거리(미학습거리, D_u)를 수학식 4에서 계산한 불확실성 추정값(S_uc)으로 나누어 제로샷 분류를 위한 이미지 벡터와 텍스트 벡터 간의 거리값을 계산하는 가공된 미학습거리(D_u')를 나타낸다.Equation (5) is the input image vector (xq) and the third text vector (y _u ) of the class not used during training

The _{processed unlearned} distance ( D _u' ).

In general, at least some unlearned images IM2 among the image data IM2 input to the object estimator 200 are more likely to be classified as a learned class rather than an unlearned class.

Unlike the embodiment of the present invention, when the unlearned distance D _u is not divided by the uncertainty estimation value S _uc , the image data IM2 input to the object estimator 200 is a class TX2 not used during learning. Even if it corresponds to -2), since the distance of the processed unlearned distance D _u ' does not decrease, the object estimator 200 classifies the image data IM2 into the class TX2-1 used for learning. can That is, the recognition performance of the object estimator 200 for the class data TX2-2 not used during learning may be deteriorated.

However, according to an embodiment of the present invention, the distance comparator 240 divides the _unlearned distance D _u by the uncertainty estimation value S _uc , and processes Since the unlearned distance D _u ' is calculated, it is possible to increase the possibility that the image input to the object estimator 200 is classified into a class not used during learning.

For example, when the image IM2 input to the uncertainty estimator 210 is highly likely to be included in the training data, the uncertainty estimation value S _uc has a relatively small value. Therefore, even if the distance comparator 240 divides the unlearned distance D _u by the uncertainty estimation value S _uc , the processed unlearned distance D _u ' is still a relatively larger value than the learning distance D _s . Therefore, the probability that the image data IM2 input to the object estimator 200 will be classified as a learned class is relatively higher than that of an unlearned class.

On the other hand, when there is a small possibility that the image IM2 input to the uncertainty estimator 210 is included in the training data, the uncertainty estimation value S _uc has a relatively large value. Accordingly, the distance comparator 240 divides the unlearned distance Du by the uncertainty estimation value S _uc , and the processed unlearned distance D _u ') value having a smaller value than the existing unlearned distance D _u . to calculate That is, the distance comparator 240 may increase the possibility that the image data IM2 input to the object estimator 200 is classified into an unlearned class.

As a result, according to an embodiment of the present invention, it is possible to improve the recognition performance of the object that the object recognition model sees for the first time, and it is possible to improve the application efficiency of the deep learning-based object recognition model in the actual industry. For example, the ability to respond to obstacles that the object recognition model in the autonomous vehicle sees for the first time may increase, and the processing performance for unidentified objects may increase in the object recognition model-based intelligent crime prevention system.

In addition, the object recognition apparatus 1000 according to an embodiment of the present invention uses a methodology of embedding an image and a text label in a common vector space, not an approach using attribute information of image data among various approaches of zero-shot learning. By doing so, the data construction cost for zero-shot learning can be reduced.

6 is a diagram illustrating an object recognition method according to an embodiment of the present invention.

Referring to FIG. 6 together with FIGS. 2 and 5 , the object recognition method according to an embodiment of the present invention includes a learning step ( S10 ) and an inference step ( S20 ). The inference step (S20) follows the learning step (S10), and in an embodiment of the present invention, the inference step (S20) and the learning step (S10) may be performed independently of each other.

First, in the learning step (S10), the first image embedding unit 110 (FIG. 2) converts the first image data IM1 into a first image vector (x _s ) and transmits it to the operation unit 140 (S11) , transmitting the text information TX1 of at least some classes stored in the dataset in the first text embedding unit 120 to the hierarchical knowledge generating unit 30 (S12), the hierarchical knowledge generating unit 130 is the upper class and Transmitting information of each of the sibling classes to the first text embedding unit 120 (S13), the first text embedding unit 110 regards the input class information as text and converts it into a first text vector (y _s ⁺ ) converting, considering the upper class information as text, converting it into a higher class vector (Y _s ^sc ), and considering the sibling class information as text and converting it into a sibling class vector (Y _s ^- ) and transmitting it to the operation unit 140 . (S14), the difference between the first image vector (x _s ), the first text vector (y _s ⁺ ), the upper class vector (Y _s ^sc ), and the sibling class vector (Y _s ^- ) received by the operation unit 140 The method includes calculating a hierarchical loss value (HSL) by calculating the Clidian distance (S15), and learning parameters such that the hierarchical loss value (HSL) is minimized (S17). The configuration corresponding to each step (S11 to S17) is the same as that described above with reference to FIGS. 1 to 4, and thus a description thereof will be omitted.

In the reasoning step (S20), the uncertainty estimator 210 classifies the second image data IM2 as separate learning data, and the distribution of probability values in which the second image data IM2 is classified into each of the classes included in the learning data. outputting (S21), calculating the uncertainty estimation value (S _uc ) using the largest value in the distribution of probability values in the uncertainty estimator 210 and transmitting it to the distance comparison unit 240 (S22), the second image The embedding unit 220 converts the second image data IM2 into a second image vector (x _q ) and transmits it to the distance comparator (S23), and the second text embedding unit 230 converts the entire class stored in the dataset Classes TX2-1 learned by the object learner 100 by considering them as text are converted into a second text vector y _s , and the classes TX2-2 that are not learned by the object learner 100 are The third text vector (y _u ) is converted to a distance comparing unit and transmitted to the distance comparator ( S24 ), where the distance comparator 240 is the distance value between the second image vector (x _q ) and the second text vector (y _s ). Calculating a temporary classification result including the learning distance (D _s ) and the unlearned distance (D _u ), which is the distance value between the second image vector (x _q ) and the third text vector (y _u ) (S25), and calculating the final classification result by the distance comparison unit 240 dividing the unlearned distance D _u by the uncertainty estimation value S _uc ( S26 ). The configuration corresponding to each step (S21 to S26) is the same as that described above with reference to FIGS. 1 and 5, and thus a description thereof will be omitted.

Although the embodiments of the present invention have been described in more detail with reference to the accompanying drawings, the present invention is not necessarily limited to these embodiments, and various modifications may be made within the scope without departing from the technical spirit of the present invention.

Therefore, the embodiments disclosed in the present invention are not intended to limit the technical spirit of the present invention, but to explain, and the scope of the technical spirit of the present invention is not limited by these embodiments. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive. The protection scope of the present invention should be construed by the following claims, and all technical ideas within the equivalent range should be construed as being included in the scope of the present invention.

1000: object recognition device
100: object learner
110: first image embedding unit
120: first text embedding unit
130: hierarchical knowledge generation unit
140: arithmetic unit
LF: hierarchical semantic loss function part
200: object estimator
210: uncertainty estimation unit
220: second image embedding unit
230: second text embedding unit
240: distance comparison department
TD: trained data
HK: Hierarchical knowledge
TI: text information

Claims (16)

an object learner for learning input first image data using hierarchical knowledge of classes stored in a dataset; and
and an object estimator that adjusts the possibility that the input second image data is classified into an unlearned class by the object learner,
The object estimator calculates an uncertainty estimation value indicating a probability that the input image data has been separately learned in advance, and uses the uncertainty estimation value to determine the possibility that the input image is classified into an unlearned class by the object learner A generalized zero-shot object recognition device that controls The method of claim 1,
The object learner,
a first image embedding unit including a neural network layer that converts the first image data into a first image vector of a specific dimension;
a first text embedding unit comprising a neural network layer for converting the text into a first text vector of a specific dimension by considering the classes stored in the dataset as text, and converting the hierarchical knowledge into a hierarchical semantic vector;
a hierarchical knowledge generating unit that receives text information of the classes from the first text embedding unit, generates the hierarchical knowledge by comparing the text information with a pre-stored hierarchical structure, and transmits the generated hierarchical knowledge to the first text embedding unit; and
and a calculator configured to receive the first image vector, the first text vector, and the hierarchical semantic vector, and calculate each Euclidean distance. 3. The method of claim 2,
The hierarchical knowledge is
Contains superclass information and sibling class information of the classes,
The object learner,
A generalized zero-shot object recognition apparatus for learning such that the first image vector and the first text vector are close to the hierarchical semantic vector of the upper class. 3. The method of claim 2,
The hierarchical knowledge is
Contains superclass information and sibling class information of the classes,
The object learner,
A generalized zero-shot object recognition apparatus that learns to distance the first image vector and the first text vector from the hierarchical semantic vector of the sibling class. 3. The method of claim 2,
The first image vector and the first text vector have the same dimension as a generalized zero-shot object recognition apparatus. 3. The method of claim 2,
The object estimator is
an uncertainty estimator comprising a neural network layer for classifying the second image data as separate learning data, and calculating the uncertainty estimation value from the distribution of the classified result value;
a second image embedding unit including a neural network layer that converts the second image data into a second image vector of a specific dimension;
A neural network layer that considers the classes learned by the object learner as text and converts them into a second text vector of a specific dimension, and considers the classes not learned by the object learner as text and converts them into a third text vector of a specific dimension a second text embedding unit including; and
and a distance comparison unit dividing the unlearned distance by the uncertainty estimate value so that the distance between the third text vector and the second image vector is reduced. 7. The method of claim 6,
The uncertainty estimation value has a lower value as the probability that the second image data is included in the training data is higher, and has a higher value as the probability that the second image data is included in the training data is low. Device. 7. The method of claim 6,
The second image vector, the second text vector, and the third text vector have the same dimension as a generalized zero-shot object recognition apparatus. 8. The method of claim 7,
The learning data is a generalized zero-shot object recognition device that is learned from the same data as the class data used by the object learner. converting the first image data into a first image vector by the first image embedding unit and transmitting the converted first image data to the calculating unit;
transmitting text information of at least some classes stored in the dataset from the first text embedding unit to the hierarchical knowledge generating unit;
transmitting, by the hierarchical knowledge generating unit, hierarchical knowledge of the class text information to the first text embedding unit;
converting the text into a first text vector of a specific dimension by the first text embedding unit considering the classes as text, and transmitting the converted text to the operation unit;
converting the hierarchical knowledge into a hierarchical semantic vector by the first text embedding unit and transmitting the converted hierarchical semantic vector to the calculating unit;
calculating, by the calculator, each Euclidean distance from the first image vector, the first text vector, and the hierarchical semantic vector to calculate a hierarchical semantic loss value; and
A generalized zero-shot object recognition method comprising learning parameters such that the hierarchical loss of meaning is minimized. 11. The method of claim 10,
Transmitting the layer knowledge to the first text embedding unit includes:
setting a hierarchical structure and storing the hierarchical knowledge generating unit; and
Comprising the step of the hierarchical knowledge generating unit generating the hierarchical knowledge by comparing the text information of the classes with the hierarchical structure,
The hierarchical knowledge is a generalized zero-shot object recognition method including upper class information and sibling class information of the classes. 12. The method of claim 11,
A generalized zero-shot object recognition method in which the hierarchical semantic loss value decreases as the distance between the first image vector and the first text vector is closer to the hierarchical semantic vector of the upper class. 12. The method of claim 11,
The hierarchical semantic loss value decreases as the distance between the first image vector and the first text vector increases from the hierarchical semantic vector of the sibling class. 12. The method of claim 11,
classifying the second image data as separate training data in the uncertainty estimator and outputting probability values that the second image data is classified into each of the classes included in the training data;
calculating an uncertainty estimate from the distribution of the probability values in the uncertainty estimator and transmitting it to the distance comparator;
converting the second image data into a second image vector by a second image embedding unit and transmitting it to a distance comparator;
converting learned classes into a second text vector by considering all classes stored in the dataset as text by a second text embedding unit, and converting unlearned classes into a third text vector;
calculating, by the distance comparator, a learning distance defined as a distance value between the second image vector and the second text vector and an unlearned distance defined as a distance between the third text vector and the second image vector; and
The generalized zero-shot object recognition method further comprising the step of dividing the unlearned distance by the uncertainty estimate by the distance comparison unit to calculate a final classification result. 15. The method of claim 14,
The uncertainty estimation value has a lower value as the probability that the second image data is included in the training data is higher, and has a higher value as the probability that the second image data is included in the training data is low. Way. 15. The method of claim 14,
The second image vector, the second text vector, and the third text vector have the same dimension as a generalized zero-shot object recognition method.

Images