KR102521184B1 - Method and system for creating synthetic training data for metric learning - Google Patents

Abstract

The present disclosure relates to a method for generating virtual learning data for metric learning. The method of generating virtual training data includes generating a plurality of embedding features of a first class corresponding to the plurality of training data of the first class by inputting a plurality of training data of a first class to an artificial neural network model; Determining a proxy of a first class based on a plurality of embedding features; inputting a plurality of training data of a second class to an artificial neural network model to obtain a plurality of embeddings of a second class corresponding to the plurality of training data of a second class; generating a feature, determining a proxy of a second class based on the plurality of embedding features of the second class, virtual embedding feature based on the plurality of embedding features of the first class and the plurality of embedding features of the second class. and generating a proxy of the virtual class based on the proxy of the first class and the proxy of the second class, wherein the virtual embedding feature belongs to the virtual class, and the virtual class belongs to the first class and the second class. different from class

Description

Virtual learning data generation method and system for metric learning {METHOD AND SYSTEM FOR CREATING SYNTHETIC TRAINING DATA FOR METRIC LEARNING}

The present disclosure relates to a method and system for generating virtual training data for metric learning, and more specifically, to a method and system for learning an artificial neural network model by generating virtual training data based on existing training data.

As artificial intelligence is applied in various fields with the recent development of artificial intelligence (AI), interest in generating an artificial neural network model with high accuracy through deep metric learning is increasing. Deep metric learning of an artificial neural network model learns a similarity metric between arbitrary data points to define an embedding space so that semantically similar data are closer to each other and different data are farther away from each other, which reduces the loss value of the loss function. It can be performed by learning an artificial neural network model in the direction.

The loss function used here can be classified into two types: pair-based loss and proxy-based loss. Pair-based loss is computed based on the similarity between data points in the embedding space. The pair-based loss type requires considering tuples (2-tuple, 3-tuple, etc.) between data points to calculate the loss, so learning is very complicated, and sampling is required to select a tuple useful for learning among multiple tuples. There is a problem with that.

On the other hand, since the proxy-based loss type considers only the relationship between an anchor and a proxy of each class, it can solve the learning complexity and sampling problem of pair-based loss and improve the performance of metric learning. However, in the case of an artificial neural network model trained only with training data, it can be over-fitted only for the seen class and lack generalization for the unseen class. there is

The present disclosure provides a method for generating virtual learning data for metric learning, a computer program stored in a recording medium, and an apparatus (system) to solve the above problems.

The present disclosure may be implemented in a variety of ways, including a method, apparatus (system) or computer program stored on a readable storage medium.

According to an embodiment of the present disclosure, a method for generating virtual learning data for metric learning corresponds to a plurality of first class learning data by inputting a plurality of first class learning data to an artificial neural network model. generating a plurality of embedding features of a first class, determining a proxy of the first class based on the plurality of embedding features of the first class, inputting a plurality of training data of a second class to an artificial neural network model, generating a plurality of embedding features of a second class corresponding to a plurality of training data of a second class; determining a proxy of a second class based on the plurality of embedding features of a second class; generating a virtual embedding feature based on an embedding feature of and a plurality of embedding features of a second class and generating a proxy of the virtual class based on the proxy of the first class and the proxy of the second class; The embedding feature belongs to a pseudo class, and the pseudo class is different from the first class and the second class.

A computer program stored in a computer readable recording medium is provided to execute the method for generating virtual learning data according to an embodiment of the present disclosure in a computer.

A system for generating virtual learning data for metric learning according to an embodiment of the present disclosure includes a memory and at least one processor connected to the memory and configured to execute at least one computer-readable program included in the memory, The at least one program inputs a plurality of training data of a first class into an artificial neural network model to generate a plurality of embedding features of a first class corresponding to the plurality of training data of the first class, and generates a plurality of embedding features of a first class. Determines a proxy of the first class based on the embedding features, inputs a plurality of training data of the second class to the artificial neural network model, and generates a plurality of embedding features of the second class corresponding to the plurality of training data of the second class. determine a proxy of the second class based on the plurality of embedding features of the second class; generate virtual embedding features based on the plurality of embedding features of the first class and the plurality of embedding features of the second class; and instructions for generating a proxy of a pseudo class based on a proxy of a class 1 and a proxy of a second class, wherein the virtual embedding feature belongs to the pseudo class, and the pseudo class is different from the first class and the second class.

In various embodiments of the present disclosure, since an artificial neural network model can be learned using additional training data considering relationships between classes, a well-structured embedding space can be constructed for the artificial neural network model. That is, it is possible to generate a robust artificial neural network model for unlearned data.

In various embodiments of the present disclosure, since a virtual class is created between existing classes, an artificial neural network model may be trained using a difficult virtual class. Accordingly, a smooth decision boundary between classes may be provided, and an artificial neural network model may provide a more generalized embedding space.

In various embodiments of the present disclosure, since a virtual class is generated using linear interpolation in an embedding space without using an additional generation network, memory and learning speed are not significantly affected.

Various embodiments of the present disclosure are directly applicable to all conventional metric learning methods using proxy-based loss.

In various embodiments of the present disclosure, it is possible to provide an artificial neural network model with higher performance than the existing metric learning method using proxy-based loss and other metric learning methods. In particular, the performance of artificial neural network models in image retrieval tasks can be greatly improved.

)를 결정하기 위한 베타 분포 함수의 파라미터(α)와 생성률(μ)의 값에 따른 인공 신경망 모델의 재현율을 나타내는 그래프이다.
도 12는 본 개시의 일 실시예에 따른 점진적으로 인공 신경망 모델을 업데이트하는 방법을 나타내는 흐름도이다.Embodiments of the present disclosure will be described with reference to the accompanying drawings described below, wherein like reference numbers indicate like elements, but are not limited thereto.
1 is a diagram illustrating an example of receiving a first image and performing image retrieval by an information processing system according to an embodiment of the present disclosure to output a second image having the highest similarity to the first image; .
2 is a schematic diagram showing a configuration in which an information processing system providing an image search service according to an embodiment of the present disclosure is connected to communicate with a plurality of user terminals.
3 is a block diagram showing internal configurations of a user terminal and an information processing system according to an embodiment of the present disclosure.
4 is a detailed block diagram of a processor according to an embodiment of the present disclosure.
5A is a diagram illustrating an example of generating one virtual class according to an embodiment of the present disclosure.
5B is a diagram illustrating an example of updating an artificial neural network model using one virtual class according to an embodiment of the present disclosure.
6A is a diagram illustrating an example of generating three virtual classes according to another embodiment of the present disclosure.
6B is a diagram illustrating an example of updating an artificial neural network model using three virtual classes according to an embodiment of the present disclosure.
7 is a flowchart illustrating a method of generating virtual learning data according to an embodiment of the present disclosure.
8 is a graph for visually comparing performance differences according to whether virtual learning data is used or not in learning/updating an artificial neural network model according to an embodiment of the present disclosure.
9 shows a t-SNE visualization result of an integrated network when an artificial neural network model is learned/updated using virtual training data generated according to an embodiment of the present invention.
10 is a graph for comparing performance differences according to whether or not virtual learning data is used in learning/updating an artificial neural network model according to an embodiment of the present disclosure with heatmap visualization results.
11 is a linear interpolation coefficient (

It is a graph showing the recall of the artificial neural network model according to the value of the parameter (α) and the generation rate (μ) of the beta distribution function for determining ).
12 is a flowchart illustrating a method of gradually updating an artificial neural network model according to an embodiment of the present disclosure.

Hereinafter, specific details for the implementation of the present disclosure will be described in detail with reference to the accompanying drawings. However, in the following description, if there is a risk of unnecessarily obscuring the gist of the present disclosure, detailed descriptions of well-known functions or configurations will be omitted.

In the accompanying drawings, identical or corresponding elements are given the same reference numerals. In addition, in the description of the following embodiments, overlapping descriptions of the same or corresponding components may be omitted. However, omission of a description of a component does not intend that such a component is not included in an embodiment.

Advantages and features of the disclosed embodiments, and methods of achieving them, will become apparent with reference to the following embodiments in conjunction with the accompanying drawings. However, the present disclosure is not limited to the embodiments disclosed below and may be implemented in various different forms, but only the present embodiments make the present disclosure complete, and the present disclosure does not extend the scope of the invention to those skilled in the art. It is provided only for complete information.

Terms used in this specification will be briefly described, and the disclosed embodiments will be described in detail. The terms used in this specification have been selected from general terms that are currently widely used as much as possible while considering the functions in the present disclosure, but they may vary according to the intention of a person skilled in the related field, a precedent, or the emergence of new technology. In addition, in a specific case, there is also a term arbitrarily selected by the applicant, and in this case, the meaning will be described in detail in the description of the invention. Therefore, the terms used in the present disclosure should be defined based on the meaning of the terms and the general content of the present disclosure, not simply the names of the terms.

Expressions in the singular number in this specification include plural expressions unless the context clearly dictates that they are singular. Also, plural expressions include singular expressions unless the context clearly specifies that they are plural. When it is said that a certain part includes a certain component in the entire specification, this means that it may further include other components without excluding other components unless otherwise stated.

Also, the term 'module' or 'unit' used in the specification means a software or hardware component, and the 'module' or 'unit' performs certain roles. However, 'module' or 'unit' is not meant to be limited to software or hardware. A 'module' or 'unit' may be configured to reside in an addressable storage medium and may be configured to reproduce one or more processors. Thus, as an example, a 'module' or 'unit' includes components such as software components, object-oriented software components, class components, and task components, processes, functions, and attributes. , procedures, subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays, or variables. Functions provided within components and 'modules' or 'units' may be combined into a smaller number of components and 'modules' or 'units', or further components and 'modules' or 'units'. can be further separated.

According to one embodiment of the present disclosure, a 'module' or 'unit' may be implemented with a processor and a memory. 'Processor' should be interpreted broadly to include general-purpose processors, central processing units (CPUs), microprocessors, digital signal processors (DSPs), controllers, microcontrollers, state machines, and the like. In some circumstances, 'processor' may refer to an application specific integrated circuit (ASIC), programmable logic device (PLD), field programmable gate array (FPGA), or the like. 'Processor' refers to a combination of processing devices, such as, for example, a combination of a DSP and a microprocessor, a combination of a plurality of microprocessors, a combination of one or more microprocessors in conjunction with a DSP core, or a combination of any other such configurations. You may. Also, 'memory' should be interpreted broadly to include any electronic component capable of storing electronic information. 'Memory' includes random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), programmable read-only memory (PROM), erasable-programmable read-only memory (EPROM), It may also refer to various types of processor-readable media, such as electrically erasable PROM (EEPROM), flash memory, magnetic or optical data storage, registers, and the like. A memory is said to be in electronic communication with the processor if the processor can read information from and/or write information to the memory. Memory integrated with the processor is in electronic communication with the processor.

In the present disclosure, 'embedding space' may refer to a space in which feature points output from the artificial neural network model by inputting input data (learning data or test data) to the artificial neural network model exist. . An 'embedding feature' may refer to a feature point in this embedding space, and each embedding feature may correspond to input data.

In the present disclosure, 'class' may refer to a category for classifying a plurality of data.

In the present disclosure, 'proxy' may refer to an embedding feature representing one class. For example, the proxy may be a median value or an average value of a plurality of embedding features included in the class. Alternatively, a proxy may be an embedding feature specified by an administrator.

In the present disclosure, an 'anchor' may refer to an embedding feature that is a reference when calculating a proxy-based loss in learning an artificial neural network model.

1 is an image search in which an information processing system 100 according to an embodiment of the present disclosure receives a first image 130 and outputs a second image 140 having the highest similarity to the first image 130. It is a diagram showing an example of performing (image retrieval). As shown, the information processing system 100 may include an artificial neural network model 110 and an image DB 120. Here, the artificial neural network model 110 may be a model obtained by learning a distance metric or a similarity metric optimized for a training data set through metric learning (or distance metric learning). That is, the artificial neural network model 110 may be trained to define an embedding space in which distances between similar images in the training data set are close (high similarity) and distances between different images are measured far (low similarity). .

In one embodiment, the information processing system 100 may receive the first image 130 . In this case, the information processing system 100 inputs the received first image 130 to the artificial neural network model 110, and has the highest similarity to the first image 130 among a plurality of images previously stored in the image DB 120. The second image 140 with high ? can be output. For example, the information processing system 100 may perform image search by outputting the second image 140 including the same character as the character included in the received first image 130 . The artificial neural network model 110 created through metric learning provides various services such as image search, person re-identification, zero-shot learning, and face recognition. can be used to do

In one embodiment, synthetic training data may be generated in order to learn the artificial neural network model 110 having a more generalized embedding space even for non-learned classes. Here, the virtual training data may include virtual embedding features and virtual proxies. By using the generated virtual training data as additional training data, the artificial neural network model 110 having a smooth decision boundary and an embedding space considering the relationship between classes can be learned. This virtual training data generation method can be applied to all metric learning using proxy-based loss, and can provide a more robust artificial neural network model for unlearned classes without significantly affecting memory and learning speed.

In FIG. 1 , the information processing system 100 receives one image and outputs one image most similar thereto, but is not limited thereto. For example, the information processing system 100 may receive one image and output n (n>2) images having the highest similarity. Alternatively, the information processing system 100 may receive a plurality of images and output one image, or receive a plurality of images and output a plurality of images. In addition, although image data is shown as input/output data of the information processing system 100 in FIG. 1, it is not limited thereto, and voice data, text data, etc. may be included in the input/output data of the information processing system 100. there is.

FIG. 2 is a schematic diagram illustrating a configuration in which an information processing system 230 providing an image search service according to an embodiment of the present disclosure is communicatively connected to a plurality of user terminals 210_1, 210_2, and 210_3. The information processing system 230 may include system(s) capable of providing various services including an image search service through the network 220, and system(s) capable of learning an artificial neural network model for providing these services. can In one embodiment, the information processing system 230 is one or more servers capable of storing, providing, and executing computer-executable programs (eg, downloadable applications) and data related to services such as image search and artificial neural network model learning. device and/or database, or one or more distributed computing devices and/or distributed database based cloud computing services.

A service such as image search provided by the information processing system 230 may be provided to a user through an image search application, a web browser application, or the like installed in each of the plurality of user terminals 210_1, 210_2, and 210_3. For example, the information processing system 230 may provide information corresponding to images received from the user terminals 210_1, 210_2, and 210_3 through an image search application or the like, or may perform a corresponding process. Additionally, the information processing system 230 may gradually learn/update the artificial neural network model using the training data set.

A plurality of user terminals 210_1 , 210_2 , and 210_3 may communicate with the information processing system 230 through the network 220 . The network 220 may be configured to enable communication between the plurality of user terminals 210_1, 210_2, and 210_3 and the information processing system 230. Depending on the installation environment, the network 220 may be, for example, a wired network such as Ethernet, a wired home network (Power Line Communication), a telephone line communication device and RS-serial communication, a mobile communication network, a wireless LAN (WLAN), It may consist of a wireless network such as Wi-Fi, Bluetooth, and ZigBee, or a combination thereof. The communication method is not limited, and the user terminals (210_1, 210_2, 210_3) as well as a communication method utilizing a communication network (eg, mobile communication network, wired Internet, wireless Internet, broadcasting network, satellite network, etc.) that the network 220 may include ), short-range wireless communication between them may also be included.

In FIG. 2, a mobile phone terminal 210_1, a tablet terminal 210_2, and a PC terminal 210_3 are illustrated as examples of user terminals, but are not limited thereto, and the user terminals 210_1, 210_2, and 210_3 may perform wired and/or wireless communication. It may be any computing device capable of this and capable of installing and executing an image-based service application, a search application, a web browser application, and the like. For example, user terminals include AI speakers, smartphones, mobile phones, navigation devices, computers, laptop computers, digital broadcasting terminals, personal digital assistants (PDA), portable multimedia players (PMPs), tablet PCs, game consoles, It may include a wearable device, an internet of things (IoT) device, a virtual reality (VR) device, an augmented reality (AR) device, a set-top box, and the like. In addition, although three user terminals 210_1, 210_2, and 210_3 are shown in FIG. 2 to communicate with the information processing system 230 through the network 220, it is not limited thereto, and a different number of user terminals may be connected to the network ( 220) to communicate with the information processing system 230.

Figure 3 is a block diagram showing the internal configuration of the user terminal 210 and the information processing system 230 according to an embodiment of the present disclosure. The user terminal 210 may refer to any computing device capable of executing image-based service applications and capable of wired/wireless communication. For example, the mobile phone terminal 210_1 of FIG. 2, the tablet terminal 210_2, It may include a PC terminal 210_3 and the like. As shown, the user terminal 210 may include a memory 312 , a processor 314 , a communication module 316 and an input/output interface 318 . Similarly, the information processing system 230 may include a memory 332 , a processor 334 , a communication module 336 and an input/output interface 338 . As shown in FIG. 3, the user terminal 210 and the information processing system 230 are configured to communicate information and/or data through the network 220 using respective communication modules 316 and 336. It can be. In addition, the input/output device 320 may be configured to input information and/or data to the user terminal 210 through the input/output interface 318 or output information and/or data generated from the user terminal 210.

The memories 312 and 332 may include any non-transitory computer readable media. According to one embodiment, the memories 312 and 332 are non-perishable mass storage devices such as random access memory (RAM), read only memory (ROM), disk drives, solid state drives (SSDs), flash memory, and the like. (permanent mass storage device) may be included. As another example, a non-perishable mass storage device such as a ROM, SSD, flash memory, or disk drive may be included in the user terminal 210 or the information processing system 230 as a separate permanent storage device separate from memory. In addition, the memories 312 and 332 may store an operating system and at least one program code (eg, a code for an image-based service application installed and driven in the user terminal 210).

These software components may be loaded from a computer readable recording medium separate from the memories 312 and 332 . A recording medium readable by such a separate computer may include a recording medium directly connectable to the user terminal 210 and the information processing system 230, for example, a floppy drive, a disk, a tape, a DVD/CD- It may include a computer-readable recording medium such as a ROM drive and a memory card. As another example, software components may be loaded into the memories 312 and 332 through a communication module rather than a computer-readable recording medium. For example, at least one program is loaded into the memories 312 and 332 based on a computer program installed by files provided by developers or a file distribution system that distributes application installation files through the network 220 . It can be.

The processors 314 and 334 may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input/output operations. Instructions may be provided to processors 314 and 334 by memories 312 and 332 or communication modules 316 and 336 . For example, processors 314 and 334 may be configured to execute instructions received according to program codes stored in a recording device such as memory 312 and 332 .

The communication modules 316 and 336 may provide configurations or functions for the user terminal 210 and the information processing system 230 to communicate with each other through the network 220, and the user terminal 210 and/or information processing. System 230 may provide configurations or functions for communicating with other user terminals or other systems (eg, separate cloud systems, etc.). For example, a request or data (eg, an image search request, etc.) generated according to a program code stored in a recording device such as a memory 312 by the processor 314 of the user terminal 210 is transmitted through the communication module 316. According to the control, it may be transferred to the information processing system 230 through the network 220 . Conversely, a control signal or command provided under the control of the processor 334 of the information processing system 230 passes through the communication module 336 and the network 220 and through the communication module 316 of the user terminal 210. It may be received by the user terminal 210 . For example, the user terminal 210 may receive information, data, and/or images related to an input image from the information processing system 230 through the communication module 316 .

The input/output interface 318 may be a means for interfacing with the input/output device 320 . As an example, the input device may include a device such as a camera, keyboard, microphone, mouse, etc. including an audio sensor and/or image sensor, and the output device may include a device such as a display, speaker, haptic feedback device, or the like. can As another example, the input/output interface 318 may be a means for interface with a device in which a configuration or function for performing input and output is integrated into one, such as a touch screen. For example, when the processor 314 of the user terminal 210 processes a command of a computer program loaded into the memory 312, information and/or data provided by the information processing system 230 or other user terminals are used. A service screen or the like configured as described above may be displayed on the display through the input/output interface 318 . Although the input/output device 320 is not included in the user terminal 210 in FIG. 3 , it is not limited thereto, and the user terminal 210 and the user terminal 210 may be configured as one device. In addition, the input/output interface 338 of the information processing system 230 is connected to the information processing system 230 or means for interface with a device (not shown) for input or output that the information processing system 230 may include. can be In FIG. 3, the input/output interfaces 318 and 338 are shown as separate elements from the processors 314 and 334, but are not limited thereto, and the input/output interfaces 318 and 338 may be included in the processors 314 and 334. there is.

The user terminal 210 and the information processing system 230 may include more components than those shown in FIG. 3 . However, there is no need to clearly show most of the prior art components. According to one embodiment, the user terminal 210 may be implemented to include at least some of the aforementioned input/output devices 320 . In addition, the user terminal 210 may further include other components such as a transceiver, a global positioning system (GPS) module, a camera, various sensors, and a database. For example, when the user terminal 210 is a smart phone, it may include components that are generally included in a smart phone, for example, an acceleration sensor, a gyro sensor, a camera module, various physical buttons, and a touch screen. Various components such as a button using a panel, an input/output port, and a vibrator for vibration may be implemented to be further included in the user terminal 210 .

According to an embodiment, the processor 314 of the user terminal 210 may be configured to operate an application providing an image-based service. At this time, codes associated with the application and/or program may be loaded into the memory 312 of the user terminal 210 . While the application and/or program is running, the processor 314 of the user terminal 210 receives information and/or data provided from the input/output device 320 through the input/output interface 318 or through the communication module 316. Information and/or data may be received from the information processing system 230 , and the received information and/or data may be processed and stored in the memory 312 . Additionally, such information and/or data may be provided to information processing system 230 via communication module 316 .

While a program for an image-based service application is running, the processor 314 is input through an input device such as a camera, microphone, or the like including a touch screen, keyboard, audio sensor, and/or image sensor connected to the input/output interface 318, or Selected text, image, video, voice, etc. may be received, and the received text, image, video, and/or voice may be stored in the memory 312 or the information processing system via the communication module 316 and the network 220. (230). In one embodiment, processor 314 provides data about an image input by a user on an image-based service application via an input device to information processing system 230 via network 220 and communication module 316. can The processor 334 of the information processing system 230 may be configured to manage, process, and/or store information and/or data received from a plurality of user terminals and/or a plurality of external systems. In one embodiment, the information processing system 230 may provide information, data, and/or images corresponding to the data on the image received from the user terminal 210 to the user terminal 210 .

4 is a detailed block diagram of a processor 334 according to an embodiment of the present disclosure. As shown, the processor 334 may include an artificial neural network model learning unit 410, an embedding feature generator 420, a proxy determiner 430, and a virtual class generator 440. The learning data DB 460 may include learning data, and the processor 334 may receive learning data from the learning data DB 460 . Learning data may be classified into a plurality of classes. For example, the learning data may include learning data associated with a first class, learning data associated with a second class, and the like. The learning data DB 460 may be included inside the information processing system or may be included in an external system or external storage device and connected to the processor 334 .

In an embodiment, the artificial neural network model learning unit 410 may receive a training data set for generating an initial artificial neural network model from the training data DB 480 and generate an initial artificial neural network model through metric learning. For example, the artificial neural network model learning unit 410 may generate an initial artificial neural network model using a proxy-based loss function based on a training data set. Here, the proxy-based loss function is all of those that calculate loss based on similarity/distance between samples (anchors) and proxies, such as Softmax, Norm-Softmax, SphereFace, CosFace, ArcFace, Proxy-NCA, SoftTriple, Proxy-Anchor, etc. It can be one of the loss functions. The purpose of metric learning is to train an artificial neural network model to place training data of the same class close to each other and to place training data of different classes far from each other. For example, in an artificial neural network model generated through metric learning, training data associated with a first class are arranged close to each other, training data associated with a second class are arranged close to each other, and training data associated with the first class and second class are placed close to each other. It is possible to build an embedding space in which training data associated with classes are spaced apart from each other. Alternatively, the processor 334 may receive an already generated initial artificial neural network model from an external system, an external storage device, or an internal storage device of the information processing system.

The embedding feature generation unit 420 receives a plurality of training data from the training data DB 460, inputs the received plurality of training data to an artificial neural network model, and generates a plurality of embedding features corresponding to each of the plurality of training data. can do. Here, the plurality of learning data may be part of the entire learning data stored in the learning data DB 460 . In an embodiment, the plurality of training data received by the embedding feature generator 420 may include a plurality of training data of a first class and a plurality of training data of a second class. The embedding feature generator 420 inputs the received plurality of training data of the first class and the plurality of training data of the second class to the artificial neural network model, and inputs the received plurality of training data of the first class corresponding to the plurality of training data of the first class. A plurality of embedding features of the second class corresponding to the embedding feature of and the plurality of training data of the second class may be generated.

In one embodiment, the proxy determiner 430 may determine a proxy of each class based on the embedding feature included in each class generated by the embedding feature generator 420 . For example, the proxy determiner 430 may determine a proxy of a first class based on a plurality of embedding features of the first class, and determine a proxy of a second class based on a plurality of embedding features of a second class. there is. Here, the proxy determiner 430 may determine a median or average value of a plurality of embedding features included in each class as a proxy of each class. A proxy may be a representative value representing each class.

The virtual class generation unit 440 may generate a virtual class including a virtual embedding feature and a proxy of the virtual class. In an embodiment, the virtual class generating unit 440 generates virtual embedding features based on the plurality of embedding features of the first class and the plurality of embedding features of the second class, and the proxy of the first class and the plurality of embedding features of the second class. Based on the proxy, you can create a proxy of the virtual class. Here, the virtual class, the first class, and the second class may be different classes.

In an embodiment, the virtual class generator 440 generates a virtual embedding feature using linear interpolation based on one of the plurality of embedding features of the first class and one of the plurality of embedding features of the second class. can create Also, the virtual class generator 440 may generate a virtual class proxy using a linear interpolation method based on the first class proxy and the second class proxy. For example, the virtual class generator 440 may generate a virtual embedding feature and a proxy of the virtual class using Equation 1 below.

는 제1 클래스의 복수의 임베딩 특징 중 하나와 제1 클래스의 프록시를 나타내고,

는 제2 클래스의 복수의 임베딩 특징 중 하나와 제2 클래스의 프록시를 나타내고,

는 가상 임베딩 특징(

)과 가상 클래스의 프록시(

)를 나타내고,

는 선형보간법 계수를 나타낸다. 수학식 1에서 확인할 수 있는 바와 같이 가상 클래스 생성부(440)는 동일한 선형보간법 계수(

)를 사용하여 가상 임베딩 특징과 가상 클래스의 프록시를 생성할 수 있다. 여기서, 사용되는 선형보간법 계수(

)는 베타 분포(beta distribution)를 이용하여 결정될 수 있다. 예를 들어, 선형보간법 계수(

)는

(여기서,

는

를 만족하는 베타 분포 함수의 파라미터)분포에 따라 결정될 수 있으며, 이렇게 결정된 선형보간법 계수(

)는 0과 1 사이의 실수일 수 있다.here,

represents one of the plurality of embedding features of the first class and a proxy of the first class,

represents one of the plurality of embedding features of the second class and a proxy of the second class,

is a virtual embedding feature (

) and the proxy of the virtual class (

),

represents the linear interpolation coefficient. As can be seen in Equation 1, the virtual class generator 440 has the same linear interpolation coefficient (

) can be used to create proxies of virtual embedding features and virtual classes. Here, the linear interpolation coefficients used (

) may be determined using a beta distribution. For example, the linear interpolation coefficient (

)Is

(here,

Is

Parameter of the beta distribution function that satisfies) may be determined according to the distribution, and the linear interpolation coefficient (

) may be a real number between 0 and 1.

According to the virtual class generation method described above, the virtual class generator 440 may generate a plurality of virtual classes. In an embodiment, the virtual class generator 440 may determine the number of virtual classes to be generated using a predetermined generation rate μ. Specifically, the virtual class generator 440 may determine the number of virtual classes to be generated by multiplying the mini-batch size by the generation rate μ. Here, the mini-batch size may represent the number of training data received by the embedding feature generator 420 from the training data DB 460 .

The artificial neural network model learning unit 410 may learn/update the artificial neural network model using the virtual embedding feature generated by the virtual class generator 440 and the virtual class including the proxy of the virtual class. In an embodiment, the artificial neural network model learning unit 410 may update the artificial neural network model using a proxy-based loss function based on the generated virtual embedding feature and the proxy of the virtual class. For example, a proxy-based loss function may calculate a loss value based on the distance between an anchor embedding feature and a proxy of each class. This proxy-based loss function can be calculated as shown in Equation 2 below.

는 학습 데이터 DB(460)로부터 수신된 학습 데이터에 대응하는 복수의 임베딩 특징(즉, 기존의 임베딩 특징)과 새로 생성된 가상 클래스의 가상 임베딩 특징을 포함하는 임베딩 특징 세트를 나타내고,

는 학습 데이터 DB(460)로부터 수신된 학습 데이터에 대응하는 복수의 임베딩 특징을 기초로 결정된 프록시(즉, 기존의 프록시)와 가상 클래스의 프록시를 포함하는 프록시 세트를 나타내고,

는 기존의 임베딩 특징-프록시 쌍과 선형보간법 계수(

)로 생성된 가상 임베딩 특징-가상 클래스의 프록시 쌍의 분포를 나타낸다. 예를 들어, 수학식 2의

는 아래의 수학식 3 내지 6 중 하나일 수 있다.here,

Represents an embedding feature set including a plurality of embedding features (i.e., existing embedding features) corresponding to the training data received from the training data DB 460 and a virtual embedding feature of a newly created virtual class,

Represents a proxy set including a proxy determined based on a plurality of embedding features corresponding to the training data received from the training data DB 460 (ie, an existing proxy) and a proxy of a hypothetical class,

is the conventional embedding feature-proxy pair and the linear interpolation coefficient (

) represents the distribution of proxy pairs of hypothetical embedding feature-hypothetical classes. For example, in Equation 2

May be one of Equations 3 to 6 below.

는

를 제외한 모든 프록시의 집합을 나타내고,

와

는 각각 a와 b사이의 유클리디안 거리와 코사인 유사도를 나타내고,

는 스케일 값을 나타낸다.here,

Is

represents the set of all proxies except

and

Represents the Euclidean distance and cosine similarity between a and b, respectively,

represents the scale value.

In one embodiment, the artificial neural network model learning unit 410 increases the distance between one of the plurality of embedding features of the first class and the proxy of the first class, and the one of the plurality of embedding features of the first class and the virtual class. The artificial neural network model may be updated such that the proxy of is farther away and one of the plurality of embedding features of the first class is farther away from the proxy of the second class. Additionally or alternatively, the artificial neural network model learning unit 410 increases the distance between the virtual embedding feature and the proxy of the virtual class, increases the distance between the virtual embedding feature and the proxy of the first class, and increases the distance between the virtual embedding feature and the proxy of the second class. The artificial neural network model can be updated so that the proxies of . Additionally or alternatively, the artificial neural network model learning unit 410 may cause a distance between one of the plurality of embedding features of the second class and a proxy of the second class to become closer, and one of the plurality of embedding features of the second class and a virtual The artificial neural network model may be updated such that the proxies of the class are distant and one of the plurality of embedding features of the second class is distant from the proxy of the first class.

5A is a diagram illustrating an example of generating one virtual class 540 according to an embodiment of the present disclosure. As shown, the first class 510 may include two embedding features 512 and 514 and one proxy 516 . Here, the proxy 516 may be a representative value representing the first class 510 . Similarly, the second class 520 includes two embedding features 522 and 524 and one proxy 526, and the third class 530 includes two embedding features 532 and 534 and one proxy ( 536) may be included.

In one embodiment, a processor (eg, at least one processor of an information processing system) may generate a virtual class using two classes among the first to third classes 510 , 520 , and 530 . For example, the processor may generate a virtual class 540 between the first class 510 and the second class 520 . In this case, virtual class 540 may include virtual embedding feature 542 and virtual proxy 544 .

Specifically, the processor selects one embedding feature 514 from among the embedding features 512 and 514 of the first class 510 and one of the embedding features 522 and 524 of the second class 520 ( Based on 524, a virtual embedding feature 542 may be generated. In one embodiment, the processor may generate virtual embedding features 542 using linear interpolation. In this case, the virtual embedding feature 542 may be positioned between the two embedding features 514 and 524 in the embedding space.

Also, the processor may generate a proxy 544 of the virtual class 540 based on the proxy 516 of the first class 510 and the proxy 526 of the second class 520 . In one embodiment, the processor may generate the proxy 544 of the hypothetical class 540 using linear interpolation. Here, the processor may generate the virtual embedding feature 542 and the proxy 544 of the virtual class 540 using the same linear interpolation coefficient. In this case, the proxy 544 of the pseudo class 540 may be located between the proxy 516 of the first class 510 and the proxy 526 of the second class 520 . The generated virtual embedding feature 542 and the proxy 544 of the virtual class 540 may constitute a virtual class 540 .

5B is a diagram illustrating an example of updating an artificial neural network model using one virtual class 540 according to an embodiment of the present disclosure. The processor may update the artificial neural network model using a proxy-based loss function based on the generated virtual embedding feature 542 and the proxy 544 of the virtual class 540 . In one embodiment, the processor may learn the artificial neural network model in a direction in which the loss value becomes small.

Specifically, the processor adjusts the anchor embedding feature 512 and the anchor embedding feature 512 so that the distance between the anchor embedding feature 512 and the proxy 516 of the first class 510 to which the anchor embedding feature 512 belongs is close. The artificial neural network model may be updated so that the distances between the proxies 526, 536, and 544 of the second class 520, the third class 530, and the virtual class 540 to which ) does not belong are increased. In FIG. 5B, one embedding feature 512 among the embedding features 512 and 514 of the first class 510 is shown as an anchor, but is not limited thereto, and the embedding features 512, 514, 522, 524, and 532 , 534) and virtual embedding features 542 may be anchor embedding features. For example, if the virtual embedding feature 542 is an anchor, the processor reduces the distance between the virtual embedding feature 542 and the proxy 544 of the virtual class 540, and the virtual embedding feature 542 and the first The artificial neural network model may be updated so that the proxies 516 , 526 , and 536 of the class 510 , the second class 520 , and the third class 530 are distant.

5A and 5B show that three existing classes 510, 520, and 530 exist in the embedding space, but are not limited thereto, and different numbers of classes may exist in the embedding space. In addition, although each class 510, 520, and 530 is illustrated as including two embedding features in FIGS. 5A and 5B, it is not limited thereto and may include any number of embedding features.

6A is a diagram illustrating an example of generating three virtual classes 640, 650, and 660 according to another embodiment of the present disclosure. The first class 610 may include two embedding features 612 and 614 and one proxy 616 . Similarly, the second class 620 includes two embedding features 622 and 624 and one proxy 626, and the third class 630 includes two embedding features 632 and 634 and one proxy ( 636) may be included.

In one embodiment, a processor (eg, at least one processor of an information processing system) may generate a plurality of virtual classes by using a combination of two classes among the first to third classes 610, 620, and 630. there is. For example, the processor may include a first pseudo class 640 between the first class 610 and the second class 620, and a second pseudo class 650 between the second class 620 and the third class 630. ), a third virtual class 660 between the first class 610 and the third class 630 may be generated.

As shown, the processor determines a first virtual embedding feature 642 based on one 614 of the first class 610 embedding features and one 624 of the second class 620 embedding features. and the proxy 644 of the first virtual class may be generated based on the proxy 616 of the first class and the proxy 626 of the second class. Similarly, the processor generates a second virtual embedding feature (652) based on one (624) of the embedding features of the second class (620) and one (632) of the embedding features of the third class (630); A proxy 654 of a second virtual class may be generated based on the proxy 626 of the second class and the proxy 636 of the third class. Similarly, the processor generates a third virtual embedding feature (662) based on one (614) of the embedding features of the first class (610) and one (634) of the embedding features of the third class (630); A proxy 664 of a third virtual class may be generated based on the proxy 616 of the first class and the proxy 636 of the third class.

6B is a diagram illustrating an example of updating an artificial neural network model using three virtual classes 640, 650, and 660 according to an embodiment of the present disclosure. The processor may update the artificial neural network model using a proxy-based loss function based on the generated virtual embedding features 642 , 652 , 662 and the proxies 644 , 654 , 664 of the virtual classes. In one embodiment, the processor may learn the artificial neural network model in a direction in which the loss value becomes small.

Specifically, the processor adjusts the anchor embedding feature 612 and the anchor embedding feature 612 so that the distance between the anchor embedding feature 612 and the proxy 616 of the first class 610 to which the anchor embedding feature 612 belongs is close. ) may update the artificial neural network model so that the distance between the proxies 626, 636, 644, 654, and 664 of a class that does not belong is increased. In FIG. 6B, one embedding feature 612 of the embedding features of the first class 610 is shown as an anchor, but is not limited thereto, and the embedding features 612, 614, 622, 624, 632, and 634 and virtual Any or all of the embedding features 642, 652, 662 may be anchor embedding features. 6A and 6B show that three existing classes 610, 620, and 630 exist in the embedding space, but are not limited thereto, and different numbers of classes may exist in the embedding space. In addition, although each class 610, 620, and 530 is illustrated as including two embedding features in FIGS. 6A and 6B, it is not limited thereto and may include any number of embedding features.

7 is a flowchart illustrating a method 700 for generating virtual learning data according to an embodiment of the present disclosure. In one embodiment, method 700 of generating virtual training data may be performed by a processor (eg, at least one processor of an information processing system). As shown, the method 700 for generating virtual training data includes a processor inputting a plurality of training data of a first class to an artificial neural network model, and a plurality of first class corresponding to the plurality of training data of the first class. It may be initiated by generating an embedding feature of (S710). Here, the artificial neural network model may be generated using a proxy-based loss function based on a training data set. Then, the processor may determine a proxy of the first class based on the plurality of embedding features of the first class (S720). For example, the proxy of the first class may be a median or average value of a plurality of embedding features of the first class.

In addition, the processor may generate a plurality of embedding features of the second class corresponding to the plurality of training data of the second class by inputting the plurality of training data of the second class to the artificial neural network model (S730). Meanwhile, the plurality of training data of the first class and the plurality of training data of the second class may be a subset of a training data set used to generate and learn an artificial neural network model. Then, the processor may determine a proxy of the second class based on the plurality of embedding features of the second class (S740). For example, the proxy of the second class may be a median or average value of a plurality of embedding features of the second class.

Then, the processor may generate virtual embedding features based on the plurality of embedding features of the first class and the plurality of embedding features of the second class (S750). In one embodiment, a virtual embedding feature may be generated using linear interpolation based on one of the plurality of embedding features of the first class and one of the plurality of embedding features of the second class. Here, the linear interpolation coefficient may be determined using a beta distribution.

Also, the processor may generate a virtual class proxy based on the first class proxy and the second class proxy (S760). In one embodiment, a proxy of a hypothetical class may be generated using a linear interpolation method based on the proxy of the first class and the proxy of the second class. In this case, the linear interpolation coefficient used may be the same as the linear interpolation coefficient used when generating the virtual embedding feature. A distance between the first class proxy and the second class proxy may be greater than a distance between the first class proxy and the pseudo class proxy and a distance between the second class proxy and the pseudo class proxy. The virtual embedding features generated according to the above-described virtual learning data generation method 700 may belong to a virtual class, and the virtual class may be different from the first class and the second class.

In one embodiment, the artificial neural network model may be updated using a proxy-based loss function based on the generated virtual embedding features and proxies of the virtual class. Here, the proxy-based loss function may calculate a loss value based on the distance between the anchor embedding feature and the proxy of each class. For example, in the proxy-based loss function, the closer the distance between the anchor embedding feature and the proxy of the class to which the anchor embedding feature belongs, the smaller the loss, and the closer the distance between the anchor embedding feature and the proxy of the class to which the anchor embedding feature does not belong. A loss value can be defined as a large loss. Specifically, the processor increases the distance between one of the plurality of embedding features of the first class and the proxy of the first class, increases the distance between one of the plurality of embedding features of the first class and the proxy of the virtual class, and The artificial neural network model may be updated so that one of the plurality of embedding features of and the proxy of the second class are distant. Additionally or alternatively, the processor may cause the artificial neural network to cause a distance between a virtual embedding feature and a proxy of a virtual class to be closer, a virtual embedding feature and a proxy of a first class to be far apart, and a virtual embedding feature and a proxy of a second class to be distant. The model can be updated.

In one embodiment, the processor receives data on the first image, and inputs the data on the received first image to the updated artificial neural network model to obtain a second image having the highest similarity to the first image among a plurality of pre-stored images. You can output data about an image.

8 is a graph for visually comparing performance differences according to whether virtual learning data is used or not in learning/updating an artificial neural network model according to an embodiment of the present disclosure. The first graph 810 represents visualization of the input space when virtual learning data is not used, the second graph 820 represents visualization of the input space when virtual learning data is used, and the third graph ( 830) represents visualization of the embedding space when virtual training data is not used, and a fourth graph 840 represents visualization of the embedding space when virtual training data is used. For this performance evaluation, a 2D isotropic Gaussian data set containing three classes of red, blue, and gray is used, and a simple feed-forward network with two-dimensional embedding for metric learning network with two-dimension embedding) and use Norm-Softmax as the proxy-based loss function. In addition, virtual learning data is generated and/or an artificial neural network model is learned/updated based on the two classes red and blue, and the gray class is not used as training data.

Through the first to fourth graphs 810, 820, 830, and 840, for both the input space and the embedding space, strict decision boundaries appear in the results when virtual learning data are not used, whereas , in the case of using virtual learning data, it can be seen that a smooth decision boundary appears that linearly transitions from the red class to the blue class. In addition, it can be confirmed that the embedding space in the case of using the virtual training data has a better location structure of embedding features than the embedding space in the case of not using the virtual training data. In addition, considering that the input data of the untrained gray class is between the red and blue classes, the artificial neural network model trained/updated using the virtual training data generated according to the embodiment of the present disclosure is linear in embedding features. It can be confirmed through the fourth graph 840 that the positional relationship is provided. Therefore, when learning/updating an artificial neural network model using virtual training data generated according to embodiments of the present disclosure, the artificial neural network model may construct an embedding space having a soft decision boundary in consideration of relationships between classes. there is.

9 shows a t-SNE visualization result of an integrated network when an artificial neural network model is learned/updated using virtual training data generated according to an embodiment of the present invention. Here, we use the CARS196 data set for training/updating the artificial neural network model. In the first graph 910, the embedding characteristics of the training data learned by the artificial neural network model and the test embedding characteristics not learned are shown, and in the second graph 920, the embedding characteristics of the same training data as the first graph 910 are shown. and proxies of virtual embedding features and virtual classes generated according to embodiments of the present disclosure. In the first graph 910 and the second graph 920, blue points are proxies of classes learned by the artificial neural network model, gray points are learned embedding features, pink points are unlearned embedding features, and red points are virtual classes. Proxies, green points represent virtual embedding features. On the other hand, in the artificial neural network model used in the first graph 910 and the second graph 920, the parameter (α) of the beta distribution function for generating virtual learning data is set to 0.4, and the generation rate (μ) is set to 1.0. It is a model trained/updated according to the training process for a batch normalized Inception network with 512 embedding dimensions.

In the first graph 910, test embeddings that have not been learned by the artificial neural network model form clusters and are located between clusters of learned embedding features. Similarly, hypothetical classes are created between the learned embedding features, as shown in the second graph 920, and these pseudoclasses function to mimic the unlearned classes. Thus, these hypothetical classes can allow the artificial neural network to additionally learn discriminative embedding features (i.e., embedding features between learned embedding features) to have better robustness to unlearned classes. there is. That is, it is preferable in metric learning to use a virtual class generated according to the embodiments of the present disclosure in order to improve robustness for a non-learned class.

Meanwhile, in order to evaluate the quantitative performance of the hypothetical class as additional training data, performance evaluation was performed by changing the configuration of the proxy and the embedding feature used for Norm-Softmax loss calculation. Table 1 below shows some or all of the existing embedding features ('Embedding-Org'), virtual embedding features ('Embedding-Syn'), existing proxies ('Proxy-Org'), and virtual proxies ('Proxy-Syn'). Indicates the recall in each case using all.

Model Embedding Proxy Recall@1 Org Syn Org Syn M1 (baseline) o o 83.3 M2 o o 83.1 M3 o o o 83.7 M4 o o o 83.7 Proxy Synthesis o o o o 84.7

Through Table 1 above, in learning/updating the artificial neural network model, when only virtual embedding features and virtual class proxies are used (M2), performance is higher than when only existing embedding features and existing proxies are used (M1). It can be seen that there is a slight decrease. This indicates that the generated virtual embedding features and virtual proxies build meaningful virtual classes in artificial neural network model training. In addition, it can be confirmed that performance is improved in the case of using the existing embedding feature, the existing proxy, and the virtual proxy (M3) as compared to the case of using only the existing embedding feature and the existing proxy (M1). This indicates that even if only the virtual proxy is used as additional training data without the virtual embedding feature, it has a positive effect on the learning of the artificial neural network model. In addition, when using virtual embedding features, existing proxies, and virtual class proxies without existing embedding features (M4), performance similar to that of existing embedding features, existing proxies, and virtual class proxies (M3) is obtained. indicate This indicates that forming a virtual class with virtual embedding features and virtual proxies can provide sufficiently differentiated information to learn/update an artificial neural network model. Finally, according to the embodiments of the present disclosure, it can be confirmed that the best performance is obtained when the existing embedding feature, the existing proxy, the virtual embedding feature, and the virtual proxy are used ('proxy synthesis').

는 기존의 클래스 i의 임베딩 특징을 나타내고,

는 기존의 클래스 i의 프록시를 나타내고,

는 기존의 클래스 i와 임의의 클래스를 기초로 생성된 가상 임베딩 특징을 나타내고,

는 기존의 클래스 i와 임의의 클래스를 기초로 생성된 가상 클래스의 프록시를 나타내고, C는 사용된 클래스의 수를 나타내고,

는 GT(ground truth) 로짓 대각선(logits diagonal)을 나타내고,

는 경쟁자(competitor) 로짓 대각선을 나타낸다.10 is a graph for comparing performance differences according to whether or not virtual learning data is used in learning/updating an artificial neural network model according to an embodiment of the present disclosure with heatmap visualization results. The first graph 1010 shows the heatmap visualization result of the cosine similarity (logit) between the embedding feature and the proxy at the 100th epoch when the artificial neural network model is trained without using a virtual class. indicate The second graph 1020 is a heat map visualization result of the cosine similarity between the embedding feature and the proxy at the 100th epoch when the artificial neural network model is trained without using the virtual class. The illustrated color bar represents reliability,

represents the embedding feature of the existing class i,

denotes a proxy of an existing class i,

Represents a virtual embedding feature generated based on an existing class i and an arbitrary class,

denotes a proxy of a virtual class generated based on an existing class i and an arbitrary class, C denotes the number of classes used,

Denotes the ground truth (GT) logits diagonal,

denotes the competitor logit diagonal.

)의 샘플링 확률을 결정하는 베타 분포 함수의 파라미터(

)에 의해 제어될 수 있다. 아래 표 2는 고정 선형보간법 계수에 따른 인공 신경망 모델의 재현율('Static')(표 2의 좌측) 및 확률적인 선형보간법 계수에 따른 인공 신경망 모델의 재현율('stochastic')(표 2의 우측)을 나타낸다.Generating a virtual class should be difficult enough for the existing training data so that the artificial neural network model can learn more discriminative training data. The difficulty values of the hypothetical class are the linear interpolation coefficients (

) parameter of the beta distribution function that determines the sampling probability of (

) can be controlled by Table 2 below shows the recall ('Static') of the artificial neural network model according to the fixed linear interpolation coefficient (left side of Table 2) and the recall rate ('stochastic') of the artificial neural network model according to the stochastic linear interpolation coefficient (right side of Table 2). indicates

Static Stochastic

Recall@1 α Recall@1 0.1 83.1 0.2 84 0.2 83.8 0.4 84.7 0.3 83.7 0.8 83.9 0.4 83.5 One 83.7 0.5 83.3 1.5 83.7

)를 0.1로 설정하는 경우에는 기존의 클래스에 너무 근접한 가상 클래스를 생성하기 때문에 인공 신경망 모델의 재현율이 낮다. 또한, 고정 선형보간법 계수(

)를 0.5로 설정하는 경우에도, 상대적으로 구별하기 쉬운 두 개의 기존의 클래스들 중간에 가상 클래스를 생성하므로 인공 신경망 모델의 재현율이 낮다. 표 2에서 고정 선형보간법 계수(

)는 기존의 클래스들과 구별하기 너무 어렵거나 너무 쉽지 않은 0.2 주변으로 설정하는 것이 바람직한 것을 확인할 수 있다.Fixed linear interpolation coefficients (

) is set to 0.1, the recall rate of the artificial neural network model is low because a virtual class that is too close to the existing class is created. In addition, the fixed linear interpolation coefficient (

) is set to 0.5, the recall of the artificial neural network model is low because a virtual class is created in the middle of two relatively easy-to-distinguish existing classes. In Table 2, the fixed linear interpolation coefficient (

) can be confirmed that it is desirable to set around 0.2, which is not too difficult or too easy to distinguish from existing classes.

In addition, through Table 2, it can be seen that the case of generating a virtual class probabilistically provides an artificial neural network model with better performance than the case of generating a virtual class statically. This is because, in the case of probabilistic generation of virtual classes, more different virtual classes having a wide range of variation can be generated. In generating the virtual class probabilistically, if the parameter (α) of the beta distribution function is set to 1.0, the beta distribution becomes equal to the uniform distribution, and if the parameter (α) of the beta distribution function is set to 1.5, the virtual class Since the class is highly likely to be generated in the middle of the two classes, the recall of the artificial neural network model is relatively low. On the other hand, if the parameter α of the beta distribution function is set to about 0.4, the probability of generating a virtual class sufficiently close to the existing class increases, and the artificial neural network model has the highest recall rate.

로 표시되는 GT의 로짓 값은 높은 예측 신뢰도로 인해 명확하게 빨간색으로 나타난다. 이는 상술한 도 8에서와 마찬가지로, 입력 공간 및 임베딩 공간에서 엄격한 결정 경계가 나타나도록 유도할 수 있으며, 과적합(overfitting) 문제를 유발할 수 있다.In order to confirm the influence of the hardness values of the virtual class, through the first graph 1010 and the second graph 1020, the embedding characteristics and Cosine similarity (logit) between proxies can be compared. The main diagonal in the first graph 1010

The logit value of GT, denoted by , is clearly red due to its high prediction reliability. This may lead to strict decision boundaries appearing in the input space and the embedding space, as in FIG. 8 described above, and may cause an overfitting problem.

및

의 GT 로짓 값은 낮은 신뢰도로 인해 노란색에서 주황색으로 나타날 수 있다. 기존의 클래스 주변에 생성된 가상 클래스가 어려운 경쟁자(hard competitor)(

및

)로 작용하여, 예측 신뢰도가 낮아지게 된다. 여기서, 제2 그래프(1020)의 주 대각선(

및

)의 신뢰도는 경쟁자의 대각선(

및

)의 신뢰도보다 높아 동일한 임베딩 특징에 대해 더 붉게(또는 붉은 색에 가깝게) 표시된다. 이러한 구성에 의해 인공 신경망 모델이 구축하는 임베딩 공간에서의 결정 경계는 보다 부드러워지며, 학습하지 않은 클래스에 대하여 보다 좋은 성능을 발휘할 수 있다.On the other hand, of the second graph 1020

and

The GT logit value of can appear yellow to orange due to low confidence. A virtual class created around an existing class is a hard competitor (

and

), the prediction reliability is lowered. Here, the main diagonal of the second graph 1020 (

and

), the reliability of the competitor's diagonal (

and

), so the same embedding features are displayed in redder (or closer to red). With this configuration, the decision boundary in the embedding space built by the artificial neural network model becomes softer, and better performance can be exhibited for unlearned classes.

In general, a well-generalized artificial neural network model provides high robustness against various input variations. Table 3 below shows the evaluation of the performance of the artificial neural network model using test data with various unlearned modifications. Through Table 3, it can be confirmed that the artificial neural network model ('PS+NS') trained using the virtual class has a high recall rate even for various input variations.

Deformation NS PS+NS Cutout 75.3 77.0 (+1.7) Dropout 59.7 62.2 (+2.5) Zoom in 64.3 65.6 (+1.3) Zoom out 78.3 80.0 (+1.7) Rotation 70.8 72.1 (+1.3) Shearing 70.3 72.0 (+1.7) Gaussian noise 65.1 67.2 (+2.1) Gaussian blur 74.4 76.3 (+1.9)

On the other hand, the artificial neural network model when the virtual class is not used shows low recall for both the training and test data sets, whereas the artificial neural network model when the virtual class is used shows the virtual class in both the training and test data sets. It shows almost twice the recall rate compared to the case without using the class. That is, the artificial neural network model using the virtual class provides a more robust embedding feature for training data and is robust for unlearned test data.

)를 결정하기 위한 베타 분포 함수의 파라미터(α)와 생성률(μ)의 값에 따른 인공 신경망 모델의 재현율을 나타내는 그래프이다. 여기서, 베타 분포 함수의 파라미터(α)는 0.2, 0.4, 0.8, 1.0, 1.5 또는 2.0으로, 생성률(μ)은 0.2, 0.5, 1.0, 1.5 또는 2.0으로 설정된다. 도 10을 참조하여 상술한 바와 같이, 베타 분포 함수의 파라미터(α)를 작은 값(예를 들어, 0.2에서 0.8사이의 값)으로 설정하는 경우, 기존의 클래스와 근접한 어려운 가상 클래스를 생성할 확률이 높아지므로 인공 신경망 모델의 재현율이 비교적 높게 나타난다. 또한 도시된 것과 같이, 생성률(μ)을 약 1.0으로 설정하는 경우에 인공 신경망 모델의 재현율이 높게 나타난다. 이는 생성률(μ)을 너무 작은 값으로 설정하면, 가상 학습 데이터가 인공 신경망 모델 학습에 충분한 영향을 미치지 못하고, 반대로 생성률(μ)을 너무 큰 값으로 설정하여 가상 클래스가 너무 많으면, 기존의 학습 데이터의 학습을 방해할 수 있기 때문이다.11 is a linear interpolation coefficient (

It is a graph showing the recall of the artificial neural network model according to the value of the parameter (α) and the generation rate (μ) of the beta distribution function for determining ). Here, the parameter (α) of the beta distribution function is set to 0.2, 0.4, 0.8, 1.0, 1.5, or 2.0, and the generation rate (μ) is set to 0.2, 0.5, 1.0, 1.5, or 2.0. As described above with reference to FIG. 10, when the parameter α of the beta distribution function is set to a small value (eg, a value between 0.2 and 0.8), the probability of generating a difficult virtual class close to the existing class As this increases, the recall rate of the artificial neural network model appears relatively high. Also, as shown, when the generation rate (μ) is set to about 1.0, the recall rate of the artificial neural network model is high. This is because if the generation rate (μ) is set too small, the virtual training data does not have a sufficient effect on the learning of the artificial neural network model, and on the contrary, if the generation rate (μ) is set too large and there are too many virtual classes, the existing training data because it can interfere with learning.

As can be seen in Table 4 below, the method of generating a virtual class according to embodiments of the present disclosure and learning/updating an artificial neural network model using the generated virtual class has a great impact on memory usage and learning speed. do not give Table 4 shows the processing time and used memory size according to the generation rate (μ). Specifically, Table 4 shows the time required to generate virtual embedding features and virtual proxies ('Generation'), time required to calculate loss ('Loss'), and embedding features and proxies when the batch size (N) is 128. Represents a number ('# of feature').

μ Time( ms ) Memory Generation Loss # of Features 0.0 - 0.558 N 0.2 0.232 0.872 (1+0.2)×N 0.5 0.233 0.873 (1+0.5)×N 1.0 0.435 1.09 (1+1.0)×N 1.5 0.449 1.3 (1+1.5)×N 2.0 0.467 1.53 (1+2.0)×N

Through Table 4, it can be seen that the processing time required to generate the pseudo class ranges from 0.232 ms to 0.467 ms, and the processing time required to calculate the loss using the generated pseudo class ranges from 0.872 ms to 1.530 ms. This is because simple linear algebra can be used to generate hypothetical classes and compute losses.

In terms of memory, when the virtual class is not used, the number of embedding features and proxies corresponds to N, and an N×N similarity matrix is generated to calculate the loss. On the other hand, when using pseudo-classes, the number of embedding features and proxies corresponds to (N + μ×N), and a similarity matrix of (N + μ×N)×(N + μ×N) is used to calculate the loss. generate That is, in the method of generating a virtual class according to the embodiments of the present disclosure and learning/updating an artificial neural network model using the generated virtual class, additional forward propagation is not required, which greatly affects memory. does not give

12 is a flowchart illustrating a method 1200 of incrementally updating an artificial neural network model according to an embodiment of the present disclosure. In one embodiment, method 1200 may be performed by a processor (eg, at least one processor of an information processing system). As shown, method 1200 may begin with a processor generating an initial artificial neural network model (S1210). For example, the processor may generate an initial artificial neural network model using a proxy-based loss function based on an initial training data set.

Then, the processor may generate virtual learning data (S1220). For example, the processor may generate virtual training data based on a subset of the training data set used to create the initial artificial neural network model. Here, the training data set may include a plurality of classes, and each of the plurality of classes may include a plurality of embedding features and proxies.

In response to generating the virtual training data, the processor may update the artificial neural network model based on the virtual training data (S1230). For example, the processor may update the artificial neural network model using the training data set used to create the artificial neural network model and the generated virtual training data. In this case, the processor may update the artificial neural network model in a direction in which the loss value decreases using the proxy-based loss function.

Then, the processor may receive additional learning data (S1240). Here, the additional training data may include at least one class, and each class may include a plurality of embedding features and proxies. The processor may update the artificial neural network model based on the received additional training data (S1250). For example, the processor may update the artificial neural network model based on the training data set used in step S1210 and/or the additional training data received in step S1240. Alternatively, the processor may update the artificial neural network model based on the training data set used in step S1210, the hypothetical training data generated in step S1230 and/or the additional training data received in step S1240. can

Thereafter, the processor may gradually learn/update the artificial neural network model by repeatedly performing steps S1220 to S1250. On the other hand, Table 5 below shows the recall rates (Recall@1 (%)) of the four benchmarks (CUB200, CARS196, SOP, In-Shop) of the artificial neural network model learned in various ways in performing image retrieval. indicates All methods use 512 embedding dimensions. In addition, in 'Net' in Table 5 below, G represents GoogleNet, and BN represents batch normalized Inception.

Method Net CUB200 CARS196 SOP In-Shop A-BIER G 57.5 - 82.0 - 74.2 - 83.1 - ABE G 60.6 - 85.2 - 76.3 - 87.3 - HTL BN 57.1 - 81.4 - 74.8 - - - Multi-Similarity BN 65.7 - 84.1 - 78.2 - 89.7 - Softmax BN 64.2 - 81.5 - 76.3 - 90.4 - PS + Softmax BN 64.9 (+0.7) 84.3 (+2.8) 77.6 (+1.3) 90.9 (+0.5) Norm-softmax BN 64.9 - 83.3 - 78.6 - 90.4 - PS + Norm-softmax BN 66.0 (+1.1) 84.7 (+1.4) 79.6 (+1.0) 91.5 (+1.1) SphereFace BN 65.4 - 83.6 - 78.9 - 90.3 - PS + SphereFace BN 66.6 (+1.2) 85.1 (+1.5) 79.4 (+0.5) 91.6 (+1.3) CosFace BN 65.7 - 83.6 - 78.6 - 90.7 - PS + CosFace BN 66.6 (+0.9) 84.6 (+1.0) 79.3 (+0.7) 91.4 (+0.7) ArcFace BN 66.1 - 83.7 - 78.8 - 91.0 - PS + ArcFace BN 66.8 (+0.7) 84.7 (+1.0) 79.7 (+0.9) 91.7 (+0.7) Proxy-NCA BN 64.3 - 82.0 - 78.1 - 90.0 - PS + Proxy-NCA BN 66.0 (+1.7) 83.1 (+1.1) 79.1 (+1.0) 91.4 (+1.4) SoftTriple BN 65.4 - 84.5 - 78.3 - 91.1 - PS + SoftTriple BN 66.6 (+1.2) 85.3 (+0.8) 79.5 (+1.2) 91.8 (+0.7) Proxy-anchor BN 68.4 - 86.1 - 79.1 - 91.5 - PS + Proxy-anchor BN 69.2 (+0.8) 86.9 (+0.8) 79.8 (+0.7) 91.9 (+0.4)

For all methods using proxy-based loss (e.g., Softmax, Norm-softmax, SphereFace, CosFace, ArcFace, Proxy-NCA, SoftTriple, Proxy-anchor), if a pseudo class is used (in Table 5, 'PS+ ') shows a higher recall rate for all benchmarks than when no pseudo class is used (items without 'PS+' in Table 5). Specifically, in fine-grained datasets with fewer categories, such as 'CUB200' and 'CARS196', the recall rate when using pseudo-classes was at least 0.7% and at most 2.8% higher, with an average of 1.2% higher. appear. In addition, in a large data set with various categories such as 'SOP' and 'In-Shop', the recall rate when using the pseudo class is at least 0.4% and at most 1.4% higher, with an average of 0.9% higher. Also, compared to methods using pair-based loss ('HTL' and 'Multi-Similarity') and ensemble methods ('A-BIER' and 'ABE'), the recall rate when using pseudo classes is Highest for all benchmarks. Considering the required learning time and amount of memory, the method of generating a virtual class and learning an artificial neural network model using the virtual class according to embodiments of the present disclosure provides high efficiency and improved performance.

The above-described method for generating virtual learning data may be provided as a computer program stored in a computer-readable recording medium to be executed on a computer. The medium may continuously store programs executable by a computer or temporarily store them for execution or download. In addition, the medium may be various recording means or storage means in the form of a single or combined hardware, but is not limited to a medium directly connected to a certain computer system, and may be distributed on a network. Examples of the medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical recording media such as CD-ROM and DVD, magneto-optical media such as floptical disks, and ROM, RAM, flash memory, etc. configured to store program instructions. In addition, examples of other media include recording media or storage media managed by an app store that distributes applications, a site that supplies or distributes various other software, and a server.

The methods, acts or techniques of this disclosure may be implemented by various means. For example, these techniques may be implemented in hardware, firmware, software, or combinations thereof. Those skilled in the art will appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein may be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchange of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the particular application and design requirements imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementations should not be interpreted as causing a departure from the scope of the present disclosure.

In a hardware implementation, the processing units used to perform the techniques may include one or more ASICs, DSPs, digital signal processing devices (DSPDs), programmable logic devices (PLDs) ), field programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, electronic devices, and other electronic units designed to perform the functions described in this disclosure. , a computer, or a combination thereof.

Accordingly, the various illustrative logical blocks, modules, and circuits described in connection with this disclosure may be incorporated into a general-purpose processor, DSP, ASIC, FPGA or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or may be implemented or performed in any combination of those designed to perform the functions described in A general purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, eg, a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other configuration.

In firmware and/or software implementation, the techniques include random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), PROM ( on a computer readable medium, such as programmable read-only memory (EPROM), erasable programmable read-only memory (EPROM), electrically erasable PROM (EEPROM), flash memory, compact disc (CD), magnetic or optical data storage device, or the like. It can also be implemented as stored instructions. Instructions may be executable by one or more processors and may cause the processor(s) to perform certain aspects of the functionality described in this disclosure.

Although the embodiments described above have been described as utilizing aspects of the presently-disclosed subject matter in one or more stand-alone computer systems, the disclosure is not limited thereto and may be implemented in conjunction with any computing environment, such as a network or distributed computing environment. . Further, aspects of the subject matter in this disclosure may be implemented in a plurality of processing chips or devices, and storage may be similarly affected across multiple devices. These devices may include PCs, network servers, and portable devices.

Although the present disclosure has been described in relation to some embodiments in this specification, various modifications and changes may be made without departing from the scope of the present disclosure that can be understood by those skilled in the art. Moreover, such modifications and variations are intended to fall within the scope of the claims appended hereto.

100, 230: information processing system
110: artificial neural network model
120: image DB
130: first image
140: second image

Claims (15)

In the virtual learning data generation method for metric learning performed by at least one processor,
generating a plurality of embedding features of a first class corresponding to the plurality of training data of a first class by inputting a plurality of training data of a first class to an artificial neural network model;
determining a proxy of a first class based on a plurality of embedding features of the first class;
generating a plurality of embedding features of a second class corresponding to the plurality of training data of a second class by inputting a plurality of training data of a second class to the artificial neural network model;
determining a proxy of a second class based on the plurality of embedding features of the second class;
generating virtual embedding features based on the plurality of embedding features of the first class and the plurality of embedding features of the second class; and
generating a proxy of a virtual class based on the proxy of the first class and the proxy of the second class;
including,
the virtual embedding feature belongs to the virtual class;
The virtual class is different from the first class and the second class, virtual learning data generation method. According to claim 1,
Generating virtual embedding features based on the plurality of embedding features of the first class and the plurality of embedding features of the second class,
Generating the virtual embedding feature using linear interpolation based on one of the plurality of embedding features of the first class and one of the plurality of embedding features of the second class
Including, virtual learning data generation method. According to claim 2,
A method for generating virtual learning data, wherein the linear interpolation coefficient is determined using a beta distribution. According to claim 3,
Generating a proxy of a virtual class based on the proxy of the first class and the proxy of the second class,
Generating a proxy of the virtual class using a linear interpolation method based on the proxy of the first class and the proxy of the second class.
including,
Wherein the virtual embedding feature and the proxy of the virtual class are generated using the same linear interpolation coefficient. According to claim 1,
The distance between the proxy of the first class and the proxy of the second class is greater than the distance between the proxy of the first class and the proxy of the virtual class and the distance between the proxy of the second class and the proxy of the virtual class. How to generate distant, hypothetical training data. According to claim 1,
Updating the artificial neural network model using a proxy-based loss function based on the generated virtual embedding feature and the proxy of the virtual class.
Further comprising, virtual learning data generation method. According to claim 6,
Wherein the proxy-based loss function calculates a loss value based on a distance between an anchor embedding feature and a proxy of each class. According to claim 7,
In the proxy-based loss function, the closer the distance between the anchor embedding feature and the proxy of the class to which the anchor embedding feature belongs, the smaller the loss, and the closer the distance between the anchor embedding feature and the proxy of the class to which the anchor embedding feature does not belong, the greater the loss. A method for generating virtual learning data that defines a loss value as According to claim 6,
Updating the artificial neural network model using a proxy-based loss function based on the generated virtual embedding feature and the proxy of the virtual class,
A distance between one of the plurality of embedding features of the first class and a proxy of the first class becomes closer, and a distance between one of the plurality of embedding features of the first class and a proxy of the hypothetical class becomes farther, and the first class Updating the artificial neural network model so that one of the plurality of embedding features of and the proxy of the second class are distant.
Including, virtual learning data generation method. According to claim 6,
Updating the artificial neural network model using a proxy-based loss function based on the generated virtual embedding feature and the proxy of the virtual class,
The artificial neural network model such that the distance between the virtual embedding feature and the proxy of the virtual class becomes closer, the virtual embedding feature and the proxy of the first class become farther away, and the virtual embedding feature and the proxy of the second class become farther apart. Steps to update
Including, virtual learning data generation method. According to claim 6,
receiving data for a first image;
inputting data for the first image into the updated artificial neural network model and outputting data for a second image having the highest similarity to the first image among a plurality of pre-stored images;
Further comprising, virtual learning data generation method. According to claim 1,
further comprising generating the artificial neural network model using a proxy-based loss function based on a training data set;
The plurality of learning data of the first class and the plurality of learning data of the second class are subsets of the learning data set. According to claim 1,
The method of generating virtual learning data, wherein the proxy of the first class is a median value or an average value of a plurality of embedding features of the first class. A computer program stored in a computer readable recording medium to execute the virtual learning data generating method according to any one of claims 1 to 13 on a computer. As a virtual learning data generation system for metric learning,
Memory; and
at least one processor connected to the memory and configured to execute at least one computer readable program contained in the memory
including,
The at least one program,
Inputting a plurality of training data of a first class to an artificial neural network model to generate a plurality of embedding features of a first class corresponding to the plurality of training data of the first class;
determine a proxy of a first class based on a plurality of embedding features of the first class;
Inputting a plurality of training data of a second class to the artificial neural network model to generate a plurality of embedding features of a second class corresponding to the plurality of training data of the second class;
determine a proxy of a second class based on the plurality of embedding features of the second class;
generate virtual embedding features based on the plurality of embedding features of the first class and the plurality of embedding features of the second class;
And instructions for generating a proxy of a virtual class based on the proxy of the first class and the proxy of the second class;
the virtual embedding feature belongs to the virtual class;
Wherein the virtual class is different from the first class and the second class.

Images