KR102446882B1 - Apparatus for recognizing faces using a large number of facial features and a learning control method thereof - Google Patents

Abstract

The present invention relates to an apparatus for recognizing a face and a method for controlling the same for efficiently learning more training data. More specifically, the present invention receives an image set for training, calculates a first embedding vector for the image set based on a face recognition model, and based on the calculated first embedding vector to perform a first learning, and a second learning based on the calculated first embedding vector and a second embedding vector, wherein the learning repeatedly performs the learning cycle of the calculating step to the second learning step, The 2 embedding vector is characterized in that it is a first embedding vector calculated in a previously performed learning cycle.

Description

Apparatus for recognizing a face using a large number of facial features and a learning control method thereof

The present invention relates to an apparatus for recognizing a face and a learning control method thereof, and more particularly, to the number of images referenced in a single learning cycle in the process of supervised learning by the apparatus for recognizing a face to recognize a face. It relates to an apparatus capable of rapidly increasing learning, as well as more accurate learning, and a learning control method thereof.

Facial recognition is a key technology in many biometric authentication applications, such as electronic payments, screen locks on smartphones, and video surveillance.

The main tasks of face recognition are divided into face verification and face identification. In face verification, a pair of faces is compared to verify whether their identities are the same. And in face identification, the identification of a given face is determined by comparing it with a pre-registered identification gallery.

Many studies have been conducted for decades to improve the accuracy of face recognition, and the recent adoption of Convolutional Neural Networks (CNN) has significantly improved the recognition accuracy. Facial recognition technologies extract feature expressions by placing a feature extraction layer behind a backbone network such as ResNet or MobileNet, which is a convolutional neural network (CNN).

A system for face recognition improves accuracy by repeatedly performing a learning (supervised learning) process to find an output that matches a given input using data with input values and corresponding output values.

The accuracy of face recognition or the time it takes to learn is affected by the amount of data set that can be handled in a single learning cycle of the learning process. If a larger amount of image data is analyzed in a single learning cycle, the facial recognition accuracy can be increased and the training time can be reduced.

However, since a lot of memory is required to extract the feature representation of an image, the amount of data that can be learned in a single learning cycle is only a small level, and simply increasing the amount of memory is not an effective approach.

Accordingly, there is a need for research on a method capable of learning a larger amount of images in a single learning cycle without depending on the amount of memory.

An object of the present invention is to provide an apparatus for recognizing a face and a learning control method thereof, which can improve overall recognition performance by accurately estimating representative features of a person based on a plurality of facial features and utilizing them.

Another problem to be solved by the present invention is to accumulate facial features while learning is in progress, to compensate for accumulated errors caused by learning, and to accurately utilize the accumulated facial features and current features to accurately represent a person's representative features ( It is to provide a device for recognizing a face capable of calculating an identity-representative vector and a learning control method thereof.

The technical problems to be achieved in the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by those of ordinary skill in the art to which the present invention belongs from the description below. will be able

According to an aspect of the present invention to achieve the above or other object, the step of receiving an image set for training (training); calculating a first embedding vector for the image set based on a face recognition model; first learning based on the calculated first embedding vector; and a second learning step based on the calculated first embedding vector and second embedding vector, wherein the learning repeatedly performs the learning cycle of the calculating step to the second learning step, and the second The embedding vector provides a learning control method of an apparatus for recognizing a face, characterized in that it is the first embedding vector calculated in a previously performed learning cycle.

(identity-representative vector)와 상기 제 1 임베딩 벡터 간의 근접도를 계산하는 단계; 및 상기 계산된 근접도가 높아지도록 파라미터를 조정하는 단계를 포함하되, 상기 조정되는 파라미터는 상기 얼굴인식 모델에 사용되는 얼굴인식 파라미터

일 수 있다.The first learning step includes an identity representative vector corresponding to a predetermined person.

calculating a proximity between an identity-representative vector and the first embedding vector; and adjusting a parameter to increase the calculated proximity, wherein the adjusted parameter is a face recognition parameter used in the face recognition model.

can be

, 상기 제 1 임베딩 벡터 및 상기 제 2 임베딩 벡터 간의 근접도를 판단하는 단계; 및 상기 판단되는 근접도가 높아지도록 상기 아이덴티티 대표 벡터

를 조정하는 단계를 포함할 수 있다.The second learning step includes an identity representative vector corresponding to a predetermined person.

, determining a proximity between the first embedding vector and the second embedding vector; and the identity representative vector to increase the determined proximity.

may include adjusting the

The second learning may further include compensating the second embedding vector, and determining the proximity may be determined based on the interpolated second embedding vector.

에 대해 현재 수행되는 학습 사이클의 아이덴티티 대표 벡터

와 기수행된 학습 사이클의 아이덴티티 대표 벡터

의 차이를 계산하는 단계를 포함하고, 상기 계산된 차이에 기초하여 보간할 수 있다.The interpolating step is a specific person

A representative vector of the identity of the learning cycle currently being performed on

and the identity representative vector of the previously performed learning cycle.

and calculating a difference between , and interpolation may be performed based on the calculated difference.

In this case, the interpolating may further include adjusting a scale to the calculated difference, and may perform interpolation based on the scaled difference.

에 의해서 계산이 이루어지며,

수학식에서

이고,

는 상기 이미지 세트,

는 상기 이미지 세트의 개수,

는 상기 제 1 임베딩 벡터,

는 레이블(labeled) 된 인물,

는 L2 노말라이즈(normalize) 된 벡터,

는

에 대한 아이덴티티 대표 벡터일 수 있다.The step of determining the proximity is a loss function defined in the following equation

Calculation is done by

in the formula

ego,

is the image set,

is the number of image sets,

is the first embedding vector,

is a labeled person,

is the L2 normalized vector,

Is

It may be an identity representative vector for .

The image set may include a plurality of images of a plurality of people, and the first embedding vector may be calculated for each of the plurality of images.

를 상기 복수 개의 인물 각각에 대응시켜 저장하는 단계를 더 포함할 수 있다.the identity representative vector

It may further include the step of storing the corresponding to each of the plurality of people.

The method may further include updating the second embedding vector so that the calculated first embedding vector is included in the second embedding vector, wherein the updated second embedding vector may be used in a learning cycle performed after the update.

According to another aspect of the present invention to achieve the above or other objects, a face recognition model that receives an image set for training and calculates a first embedding vector for the input image set; a first learning unit performing first learning based on the calculated first embedding vector; and a second learning unit configured to perform a second learning based on the calculated first embedding vector and the second embedding vector, wherein the learning repeatedly performs a learning cycle of the calculation, the first and the second learning, and the second learning is performed. The second embedding vector may be the first embedding vector calculated in a previously performed learning cycle.

(identity-representative vector)와 상기 제 1 임베딩 벡터 간의 근접도를 계산하고, 상기 계산된 근접도가 높아지도록 파라미터를 조정하되, 상기 조정되는 파라미터는 상기 얼굴인식 모델에 사용되는 얼굴인식 파라미터

일 수 있다.The first learning unit, an identity representative vector corresponding to a predetermined person

Calculate a proximity between an identity-representative vector and the first embedding vector, and adjust a parameter to increase the calculated proximity, wherein the adjusted parameter is a face recognition parameter used in the face recognition model

can be

, 상기 제 1 임베딩 벡터 및 상기 제 2 임베딩 벡터 간의 근접도를 판단하고, 상기 판단되는 근접도가 높아지도록 상기 아이덴티티 대표 벡터

를 조정할 수 있다.The second learning unit, an identity representative vector corresponding to a predetermined person

, determine the proximity between the first embedding vector and the second embedding vector, and the identity representative vector so that the determined proximity increases

can be adjusted.

When determining the proximity, the second learner may interpolate the second embedding vector and determine based on the interpolated second embedding vector.

에 대해 현재 수행되는 학습 사이클의 아이덴티티 대표 벡터

와 기수행된 학습 사이클의 아이덴티티 대표 벡터

의 차이를 계산하고, 상기 계산된 차이에 기초하여 보간할 수 있다.In the interpolation of the second learning unit, a specific person

A representative vector of the identity of the learning cycle currently being performed on

and the identity representative vector of the previously performed learning cycle.

may calculate a difference between , and interpolate based on the calculated difference.

In the interpolation, the second learner may adjust a scale to the calculated difference and perform interpolation based on the scaled difference.

에 의해서 계산이 이루어지며,

The first and second learning units determine the proximity, a loss function defined in the following equation

Calculation is done by

이고,

는 상기 이미지 세트,

는 상기 이미지 세트의 개수,

는 상기 제 1 임베딩 벡터,

는 레이블(labeled) 된 인물,

는 L2 노말라이즈(normalize) 된 벡터,

는

에 대한 아이덴티티 대표 벡터일 수 있다.in the formula

ego,

is the image set,

is the number of image sets,

is the first embedding vector,

is a labeled person,

is the L2 normalized vector,

Is

It may be an identity representative vector for .

The image set may include a plurality of images of a plurality of people, and the first embedding vector may be calculated for each of the plurality of images.

를 상기 복수 개의 인물 각각에 대응시켜 저장하는 분류부를 더 포함할 수 있다.the identity representative vector

It may further include a classification unit for storing the corresponding to each of the plurality of people.

Further comprising a queue storage unit for updating the second embedding vector so that the calculated first embedding vector is included in the second embedding vector, the updated second embedding vector may be used in a learning cycle performed after the update .

The effect of the device for recognizing a face and the learning control method thereof according to the present invention will be described as follows.

According to at least one of the embodiments of the present invention, there is an advantage that the number of images used for learning for face recognition can be dramatically increased while appropriately maintaining memory usage.

In addition, according to at least one of the embodiments of the present invention, there is an advantage that a learning process for face recognition can be performed more quickly and accurately through a learning process that comprehensively considers a large number of images.

Further scope of applicability of the present invention will become apparent from the following detailed description. However, since various changes and modifications within the spirit and scope of the present invention can be clearly understood by those skilled in the art to which the present invention pertains, the specific embodiments described in the detailed description are given by way of example only. should be understood

1 shows a conceptual diagram of a face recognition model 110 for extracting a feature expression (features of a face) based on a convolutional neural network.
2 is a conceptual diagram illustrating a classification step operation of a general classifier 210 .
3 is a diagram illustrating one cycle of a general supervised learning process.
4 is a diagram illustrating a block diagram of a face recognition apparatus 100 according to an embodiment of the present invention.
5 is a diagram illustrating one cycle of a supervised learning process according to an embodiment of the present invention.
6 is a diagram illustrating a flowchart of the face recognition apparatus 100 according to an embodiment of the present invention.
7 and 8 are diagrams illustrating a conceptual diagram of a method for interpolating an embedding vector according to an embodiment of the present invention.
9 is experimental data that proves the effect of interpolation according to an embodiment of the present invention.
10 is a diagram comparing an increase in memory required as the amount of learning data increases.
11 shows experimental data when the face recognition method according to an embodiment of the present invention is applied to an image search technology.

Hereinafter, the embodiments disclosed in the present specification will be described in detail with reference to the accompanying drawings, but the same or similar components are assigned the same reference numerals regardless of reference numerals, and redundant description thereof will be omitted. The suffixes "module" and "part" for components used in the following description are given or mixed in consideration of only the ease of writing the specification, and do not have distinct meanings or roles by themselves. In addition, in describing the embodiments disclosed in the present specification, if it is determined that detailed descriptions of related known technologies may obscure the gist of the embodiments disclosed in this specification, the detailed description thereof will be omitted. In addition, the accompanying drawings are only for easy understanding of the embodiments disclosed in this specification, and the technical idea disclosed herein is not limited by the accompanying drawings, and all changes included in the spirit and scope of the present invention , should be understood to include equivalents or substitutes.

Terms including ordinal numbers such as first, second, etc. may be used to describe various elements, but the elements are not limited by the terms. The above terms are used only for the purpose of distinguishing one component from another.

When a component is referred to as being “connected” or “connected” to another component, it may be directly connected or connected to the other component, but it is understood that other components may exist in between. it should be On the other hand, when it is said that a certain element is "directly connected" or "directly connected" to another element, it should be understood that the other element does not exist in the middle.

The singular expression includes the plural expression unless the context clearly dictates otherwise.

In the present application, terms such as “comprises” or “have” are intended to designate that a feature, number, step, operation, component, part, or combination thereof described in the specification exists, but one or more other features It is to be understood that this does not preclude the possibility of the presence or addition of numbers, steps, operations, components, parts, or combinations thereof.

* Justice

- Instance: means one image (also called sample).

- Identity: It means a single person (expressed as a class in a normal image search (Vision Task)).

- Face recognition model (Model, encoder): It is a trained Convolutional Neural Network (CNN) that collectively refers to the structure and model parameters. When a face recognition model is given an image, it is encoded into an embedding vector by reflecting the features of the given image. Through the training process, the model parameters are optimized. As an example, facial features included in an image may be mapped to vectors using FaceNet or ResNet.

- Classifier: When an embedding vector for a specific image (a face included in) is obtained through the face recognition model, the 'identity representative vector' closest to (high similarity) to the embedded vector obtained from the database is determined It is a configuration that identifies identities (persons) through In the learning phase, the classifier model continuously updates the parameters of the stored 'identity representative vector' according to the result of the loss function.

- Learning (training or learning): refers to the process for optimizing the model, and optimizes the parameters of the face recognition model or the classifier model based on tens of thousands or more images. Due to the physical limitations of the computing device, it is not possible to learn all the images at once, and the learning is performed in a manner of iteration in units of an image set (mini-batch) of a preset size.

- Embedding Vector: When an input is given, it is a feature vector that embeds the meaning of the input image using the trained model. The image may be encoded into an embedding vector by the face recognition model. Similarity between images can be determined by calculating the distance between each embedding vector (if they are similar, the distance between the vectors is close, and the distance between the vectors increases as the similarity decreases). For example, when an image is input to FaceNet or ResNet, the output output can be called an embedding vector.

- Embedding Space: A virtual space to which embedding vectors are mapped.

- Identity-representative vector: As a weight vector/weight parameter of a classifier, it refers to a vector representing a specific identity (person). Semantically, it is the average of a plurality of embedding vector(s) of instances belonging to a specific identity (implementationally, it means the weight vector of the last fully-connected layer). For example, a vector representing the embedding vectors for the 'Donald Trump' photos (for example, the average of the embedding vectors) may be called an 'identity representative vector' for the person named 'Donald Trump'. The identity representative vector is optimized by the learning process.

- Loss function (or cost function): A function for calculating how much cost or loss is, and to calculate the degree of dissimilarity between embedding vectors (or embedding vectors vs. identity representative vectors) is a function for In the present invention, if the difference between the embedding vector and the identity representative vector is large, the loss is large, and if the difference is small, the loss is small. The optimization of parameters (facial recognition model parameters or classifier parameters) is made in the direction of minimizing the result calculated by the loss function.

- Batch: refers to the entire instance (image) data set (group) used for supervised learning of the model.

- Mini-batch: A data set (group) to be processed (eg, a single learning cycle) as one during supervised learning of a model. For example, when 210 instances (images) are input as training data of a single learning cycle (1 iteration) as an input of a face recognition model, a set of 210 images is called a mini-batch.

1 shows a conceptual diagram of a face recognition model 110 (hereinafter referred to as a model) for extracting feature expressions (features of a face) based on a convolutional neural network.

When an image for a specific person (identity) is input, the model 110 encodes a facial feature expression of the person. In this case, the result of the encoding performed may be expressed as an N-dimensional vector. The vector expressed in this way reflects the characteristics of the corresponding face, and is called an embedding vector.

In FIG. 1 , when a first image 101-1 for a first person 105-1 is input to the model 110, the model 110 converts the first image 101-1 into a first embedding vector 102 -1) to encode. Similarly, the second image 101-2 of the first person 105-1 is encoded with the second embedding vector 102-2, and the third image 101- of the second person 105-2 is encoded. 3) is encoded into the third embedding vector 102-3. In this case, the first and second people 105-1 and 105-2 are different people, and the first and second images 101-1 and 101-2 are the same for the first person 105-1. Assume it is a different image.

When checking the distance (or proximity) between the embedding vectors obtained in this way, it is possible to check whether the same person or a different person. When the distance between the first to third embedding vectors 102-1 to 102-3 is calculated, the first and second embedding vectors 102-1 and 102-2 are relatively close to each other and the third embedding vector 102- 3) can be confirmed as being relatively far away. Accordingly, the persons included in the first and second images 101-1 and 101-2 are the same person (first person), and the person included in the third image 101-3 is different (second person). person) can be identified.

After all, when checking the embedding vectors obtained through the model 110 , it is possible to check whether they are the same person by checking whether the distance is relatively close or not.

The model 110 can be expressed by Equation 1 as follows.

는 얼굴인식 모델을 나타내는 함수,

는 임베딩 벡터,

는 모델(110)에 사용되는 얼굴인식 파라미터이고,

는 입력되는 이미지를 의미한다. 얼굴인식 파라미터

는 수 만장 이상의 이미지 데이터에 대한 지도 학습(supervised learning)을 통하여 획득한 파라미터를 의미하며, 지도 학습 과정에서 지속적으로 최적화되는 방향으로 업데이트된다. 이때 지도 학습이란, 입력값과 그에 따른 출력값이 있는 데이터를 이용하여 주어진 입력에 맞는 출력을 찾는 학습을 의미하며, 얼굴인식에 관한 학습에서는 얼굴이 포함되어 있는 이미지와 해당 얼굴이 누구인지 여부를 알고 있는 상태(정답을 알고 있는 상태)에서 이루어지는 학습을 의미한다.here

is a function representing the face recognition model,

is the embedding vector,

is a face recognition parameter used in the model 110,

is the input image. face recognition parameters

is a parameter obtained through supervised learning of tens of thousands of image data, and is continuously updated in the direction of optimization in the supervised learning process. In this case, supervised learning means learning to find an output that matches a given input using data with input values and corresponding output values. It means learning that takes place in the state of being (the state of knowing the correct answer).

2 is a conceptual diagram illustrating a learning stage of a general classifier 210 .

(Identity-representative Vector)'라 한다. 분류부(210)는 아이덴티티 대표 벡터

역시 상술한 지도 학습 과정(도 3을 통하여 후술)을 통하여 최적화되는 방향으로 지속적으로 업데이트(

)한다.The classification unit 210 may hold a representative embedding vector for each person. These representative embedding vectors are referred to as 'identity representative vectors.

(Identity-representative Vector)'. Classification unit 210 is an identity representative vector

Also continuously updated (

)do.

와 얼굴인식 파라미터

를 최적화되는 방향으로 지속적으로 다듬는 구체적인 지도 학습 과정에 대해서 설명한다.Referring to FIG. 3 below, identity representative vector

and face recognition parameters

The detailed supervised learning process of continuously refining in the direction of optimization will be described.

3 is a diagram illustrating one cycle of a general supervised learning process.

As described above, supervised learning refers to learning to find an output in a state in which an output value is given together with an input value. That is, supervised learning is performed in a state in which information (output value) about the person included in the image is given together with an image (input value). In this way, the data input by forming a pair of an input value and an output value is called training data.

가 최적화될 수 있다. 즉, 학습 데이터가 많으면 많을수록 더 정확한 모델(110) 및 아이덴티티 대표 벡터

를 얻을 수 있다. 일반적으로 얼굴인식 기술에 사용되는 학습 데이터 세트의 경우 수십만에 달하는 사람들에 대해 수백만 장의 이미지가 이용된다.As the learning using the learning data is made, the model 110 and the identity representative vector

can be optimized. That is, the more training data, the more accurate the model 110 and identity representative vector.

can get For training data sets typically used in facial recognition technology, millions of images are used for hundreds of thousands of people.

Therefore, it is preferable to base the learning data as much as possible upon learning, but due to the physical limitations of the memory provided in the device, the number of learning data that can be input at one time for learning is inevitably limited. This is because a lot of memory is required when the model 110 encodes an input instance (image).

를 최적화시키는 것이 일반적이다. 즉 학습 데이터들을 미니 배치 단위로 반복적으로 학습하면서 얼굴인식 장치를 최적화시킨다. 예를 들어 일반적으로 약 210개의 인스턴스(이미지)가 한 번의 학습 데이터로 입력된다.Therefore, considering the available range of memory, the entire training data set (batch) is split into small data sets (mini-batch), and while these mini-batches are sequentially and iteratively trained, the model 110 and the identity representative vector

It is common to optimize That is, the face recognition device is optimized while repeatedly learning the training data in mini-batch units. For example, in general, about 210 instances (images) are input as training data at one time.

로 인코딩하고, 학습부(310)로 전달한다. 학습부(310)는 임베딩 벡터

가 대응되는 아이덴티티 대표 벡터

와 가장 가까워지고, 대응되지 않는 아이덴티티 대표 벡터

와는 멀어지도록 모델(110)의 파라미터

와 아이덴티티 대표 벡터

를 최적화시킨 후 각각 모델(110) 및 분류부(210)에 업데이트시키는 방식으로 학습을 수행한다. 즉, 동일 인물에 대한 임베딩 벡터

와 아이덴티티 대표 벡터

는 가까워지고, 다른 인물에 대한 임베딩 벡터

와 아이덴티티 대표 벡터

는 멀어지도록 최적화시킨다는 것이다.Referring to the illustrated figure, the mini-batch 301 is input to the model 110 (eg, about 210 images). The model 110 embeds the input mini-batch 301 into an embedding vector.

is encoded and transmitted to the learning unit 310 . Learning unit 310 embedding vector

Identity representative vector that corresponds to

Identity representative vector that is closest to and does not correspond to

parameters of the model 110 so as to be far from

and identity representative vector

After optimizing , learning is performed by updating the model 110 and the classifier 210, respectively. That is, embedding vectors for the same person

and identity representative vector

is getting closer, embedding vectors for other figures

and identity representative vector

is to optimize away from it.

와 아이덴티티 대표 벡터

간에 얼마나 가깝거나(근접도) 얼마나 먼지(손실도)를 아래 수학식 2의 손실 함수(Loss function)

에 기초하여 산출할 수 있다.At this time, the learning unit 310 embedding vector

and identity representative vector

How close (proximity) or how much dust (loss) is the loss function of Equation 2 below

can be calculated based on

는 아래 수학식 3으로 표현될 수 있다.here

can be expressed by Equation 3 below.

는 미니 배치(301)로 입력된 이미지 세트,

는 상기 이미지 세트의 개수,

는 임베딩 벡터,

는 레이블(labeled) 된 아이덴티티(즉 학습 데이터의 결과값),

는 L2 노말라이즈(normalize) 된 벡터,

는

에 대한 아이덴티티 대표 벡터이다. 그리고

는 데이터 베이스에 저장되어 있는 아이덴티티 대표 벡터 집단을 의미한다.

is the image set input as mini-batch 301,

is the number of image sets,

is the embedding vector,

is the labeled identity (that is, the result value of the training data),

is the L2 normalized vector,

Is

It is an identity representative vector for . and

denotes a group of identity representative vectors stored in the database.

에 의해서 계산된 '손실'이 최소화되는 방향으로 파라미터

와 아이덴티티 대표 벡터

를 수정하고, 수정된 값으로 업데이트하도록 수정된 파라미터

를 모델(110)에, 수정된 아이덴티티 대표 벡터

분류부(210)에 회신하는 방식으로 학습을 수행한다.That is, the learning unit 310 is a loss function

parameter in the direction in which the 'loss' calculated by

and identity representative vector

Modified parameters to modify and update to the modified values

In the model 110, the modified identity representative vector

Learning is performed by replying to the classification unit 210 .

와 아이덴티티 대표 벡터

는 해당 학습 사이클 단계에서 최적화된 결과일 것이다. 전체 학습 데이터 중 나머지 학습 데이터에 대해서도 마찬가지로 상술한 학습 싸이클이 반복적으로 수행될 수 있으며, 학습 싸이클이 반복됨에 따라 파라미터

와 아이덴티티 대표 벡터

역시 반복적으로 최적화가 이루어진다.This updated parameter

and identity representative vector

will be the optimized result in the corresponding learning cycle step. Similarly, the above-described learning cycle may be repeatedly performed for the remaining training data among the entire training data, and as the learning cycle is repeated, the parameters

and identity representative vector

Optimization is also iteratively performed.

As described above, since the conventional learning method uses a mini-batch, it is difficult to handle a large amount of training data (identities or instances) at a time. The small size of training data that can be trained at one time means that it takes many iterations to check tens of thousands of identities, and this requirement requires that all identities be considered comprehensively while determining the optimal decision boundary in the embedding space. It complicates the task of learning the boundary). This is because, when learning is repeatedly performed in mini-batch units as described above, the longer the repetition is accumulated, the more the initial learning result is not properly reflected. Learning at the beginning of iteration is rarely reflected later. Therefore, even if learning is repeated for a long time, the learning accuracy does not rise above a certain level and becomes saturated.

Simply increasing the size of the mini-batch may alleviate some problems, but it is impractical due to memory limitations. Also, simply increasing the size of the mini-batch does not guarantee improved accuracy.

Therefore, in the present invention, it is intended to propose a method that can comprehensively consider a vast amount of identities without relying on memory. To this end, the present invention proposes to perform learning in consideration of the embedding vector (the first embedding vector) for the mini-batch and the embedding vector (the second embedding vector) used in the past learning process.

Embodiments of the present invention will be described in detail below with reference to the drawings.

4 is a diagram illustrating a block diagram of a face recognition apparatus 100 according to an embodiment of the present invention. 5 is a diagram illustrating one cycle of a supervised learning process according to an embodiment of the present invention. 6 is a diagram illustrating a flowchart of the face recognition apparatus 100 according to an embodiment of the present invention. Hereinafter, it will be described with reference to FIGS. 4 to 6 together.

The face recognition apparatus 100 according to an embodiment of the present invention may be configured to include a model 110 , a classification unit 210 , a learning unit 310 , and a queue storage unit 410 . The components shown in FIG. 4 are not essential in implementing the face recognition apparatus 100, so the face recognition apparatus 100 described in this specification may include more or fewer components than those listed above. can have

When an image of a specific person (identity) is input, the model 110 extracts a facial feature expression of the person and performs encoding based on the extracted feature expression. In this case, the result of the encoding performed may be expressed as an N-dimensional vector. The vector expressed in this way reflects the characteristics of the corresponding face, and is called an embedding vector.

(Identity-representative Vector)'를 보유한다. 그리고 임베딩 벡터가 입력되면, 분류부(210)는 가장 유사도가 높은 아이덴티티 대표 벡터를 매칭시키는 방식으로 입력된 임베딩 벡터를 분류한다.The classification unit 210 is an 'identity representative vector for each of several people (identities).

(Identity-representative Vector)'. And when an embedding vector is input, the classification unit 210 classifies the input embedding vector by matching the identity representative vector with the highest similarity.

와 아이덴티티 대표 벡터

를 최적화(optimize)시키는 방향으로 업데이트한다. 본 발명의 일실시에에 따른 학습부(310)는 얼굴인식 파라미터

를 최적화시키기 위한 제 1 학습부(310-1)와 아이덴티티 대표 벡터

를 최적화시키기 위한 제 2 학습부(310-2)를 포함하도록 구성될 수 있다. 제 1 및 제 2 학습부(310-1, 310-2)에 대해서는 이하 좀 더 상세히 후술하기로 한다.The learning unit 310 performs learning using the input learning data, and as a result of performing the learning, the face recognition parameter

and identity representative vector

is updated in the direction of optimizing The learning unit 310 according to an embodiment of the present invention is a face recognition parameter

The first learning unit 310-1 and the identity representative vector for optimizing

It may be configured to include a second learning unit 310 - 2 for optimizing . The first and second learning units 310-1 and 310-2 will be described in more detail below.

The queue storage unit 410 (Queue storage) accumulates and stores the embedding vector and the identity representative vector used in each learning cycle. That is, in the queue storage unit 410, the embedding vectors and identity representative vectors used in the past learning process are accumulated and stored. In the present invention, in this way, the embedding vector used in the past learning process, stored in the queue storage unit 410, is learned together with the embedding vector for the mini-batch input in the current learning cycle, and a vast amount of identities (instances) ) is proposed to be considered comprehensively.

(501)로 인코딩(S602)한다. 제 1 학습부(310-1)는 상기 제 1 임베딩 벡터

(501)에 기초하여 제 1 학습을 수행(S603)한다. 이때 미니 배치(301)는 복수 개의 아이덴티티에 대한 복수 개의 이미지를 포함할 수 있다. 그리고 상기 인코딩되는 제 1 임베딩 벡터(들)

는 상기 복수 개의 이미지 각각에 대해서 산출될 수 있을 것이다.The learning process of FIG. 5 will be described with reference to the flowchart of FIG. 6 . As shown in FIG. 5 , when the mini-batch 301 is input ( S601 ), the model 110 converts the input mini-batch 301 into the first embedding vector(s).

(501) is encoded (S602). The first learning unit 310-1 is the first embedding vector.

Based on (501), the first learning is performed (S603). In this case, the mini-batch 301 may include a plurality of images for a plurality of identities. and the encoded first embedding vector(s)

may be calculated for each of the plurality of images.

(identity-representative vector)와 상기 제 1 임베딩 벡터

가 서로 근접해 지도록(가까워지도록) 얼굴인식 파라미터

를 수정하는 방식으로 업데이트하는 것을 의미한다. 수정된 얼굴인식 파라미터

는 모델(110)로 전달되어, 업데이트가 모델(110)에 반영될 수 있을 것이다. 즉, 제 1 학습 단계에서는 얼굴인식 파라미터

가 최적화(S604)된다.Here, the first learning is an identity representative vector for the same identity.

(identity-representative vector) and the first embedding vector

face recognition parameters so that (closer) to each other

means to update it in a way that fixes it. Modified face recognition parameters

is transmitted to the model 110 , so that the update may be reflected in the model 110 . That is, in the first learning step, the face recognition parameters

is optimized (S604).

(identity-representative vector)와 상기 제 1 임베딩 벡터

간의 근접도를 판단하는 단계 및 상기 판단되는 근접도가 높아지도록 파라미터를 최적화(optimize)시키는 단계를 포함할 수 있다.To this end, the first learning step according to an embodiment of the present invention is an identity representative vector

(identity-representative vector) and the first embedding vector

It may include determining the proximity between the two, and optimizing a parameter so that the determined proximity increases.

And in the step of determining the proximity, the first loss function of Equation 4 may be used. And also in the optimization step, the optimization may be made in a direction in which the loss calculated through the first loss function is minimized.

는 미니 배치(301)로 입력된 이미지 세트,

는 미니 배치(301)로 입력된 이미지 세트의 개수,

는 제 1 임베딩 벡터이다.here

is the image set input as mini-batch 301,

is the number of image sets input into the mini-batch 301,

is the first embedding vector.

(502)를 로드(S605)하고, 제 2 학습을 수행(S606)한다. 제 2 학습부(310-2)는 제 1 대기열저장부(410-1)에 저장되어 있던 제 2 임베딩 벡터

(502)와 상기 제 1 임베딩 벡터

와 함께 제 2 학습(S606)을 수행한다.Furthermore, the learning unit 310 according to an embodiment of the present invention uses the second embedding vector to use the past data stored in the queue storage unit 410 together.

(502) is loaded (S605), and the second learning is performed (S606). The second learning unit 310-2 is a second embedding vector stored in the first queue storage unit 410-1.

502 and the first embedding vector

A second learning ( S606 ) is performed together with .

, 상기 제 1 임베딩 벡터

및 제 2 임베딩 벡터

가 서로 근접해 지도록(가까워지도록) 아이덴티티 대표 벡터

를 수정하는 방식으로 업데이트할 수 있다. 수정된 아이덴티티 대표 벡터

는 분류부(210)로 전달되어, 업데이트가 분류부(210)에 반영될 수 있을 것이다. 제 2 학습 단계에서는 아이덴티티 대표 벡터

, 즉 분류부(210)의 파라미터가 최적화된다.The second learning step according to an embodiment of the present invention is similar to the first learning step described above. The second learning step is an identity representative vector for the same identity.

, the first embedding vector

and a second embedding vector

identity representative vector so that (closer) to each other

can be updated by modifying the . Modified Identity Representative Vector

is transmitted to the classification unit 210 , and the update may be reflected in the classification unit 210 . In the second learning stage, the identity representative vector

That is, the parameters of the classification unit 210 are optimized.

(identity-representative vector)와 상기 제 1 임베딩 벡터

및 제 2 임베딩 벡터

간의 근접도를 판단하는 단계 및 상기 판단되는 근접도가 높아지도록 파라미터를 최적화(optimize)시키는 단계를 포함할 수 있다.To this end, the second learning step according to an embodiment of the present invention is an identity representative vector

(identity-representative vector) and the first embedding vector

and a second embedding vector

It may include determining the proximity between the two, and optimizing a parameter so that the determined proximity increases.

And in the step of determining the proximity, the second loss function of Equation 5 may be used. And also in the optimization step, the optimization may be made in a direction in which the loss calculated through the second loss function is minimized.

는 제 1 대기열저장부(410-1)에 저장되어 있는 제 2 임베딩 벡터

(들)을 의미한다. 수학식 4와 공통되는 설명은 생략한다.

is the second embedding vector stored in the first queue storage unit 410-1

means (s). Descriptions common to Equation 4 will be omitted.

(분류부의 파라미터)가 최적화될 수 있다. 즉, 과거 학습 데이터는 새롭게 인코딩되지 않기 때문에(과거 이미 인코딩된 임베딩 벡터를 그대로 사용) 메모리 사용을 크게 늘리지 않으면서도, 더 많은 학습 데이터를 참고하여 학습이 가능하다는 것이다.That is, according to the above-described second learning step, not only the learning data for the mini-batch 301 input in the current learning cycle, but also the learning data used for the past learning are comprehensively considered together, so that the identity representative vector

(parameters of the classifier) can be optimized. In other words, since the past learning data is not newly encoded (using the embedding vector encoded in the past as it is), it is possible to learn by referring to more training data without significantly increasing memory usage.

를 제 2 대기열저장부(410-2)에 누적 저장할 수 있다.The facial recognition apparatus 100 is updated (S608) based on the parameters optimized by the first and second learning steps. Then, the first embedding vector in the current learning cycle is accumulated in the first queue storage unit 410-1 (added to the current second embedding vector) to be used again in the subsequent learning process, and the identity representative vector

may be accumulated and stored in the second queue storage unit 410 - 2 .

Subsequently, it moves to step S609 to determine whether learning has been performed on the entire training data, and if the learning is not completed, returns to step S601 again to receive an input of another mini-batch set.

를 학습에 반영하는데 있어서, 그대로 반영하지 않고 보간(compensate)시킨 후 반영하도록 제안한다. 이는 과거에 학습된 임베딩 벡터들은 과거의 파라미터에 의해서 인코딩된 상태로서, 그동안 이루어진 파라미터의 업데이트를 반영하지 않고 있기 때문이다.Further, in another embodiment of the present invention, the second embedding vector stored in the first queue storage unit 410-1

It is proposed to reflect after interpolation (compensate) instead of reflecting it as it is in reflecting it in learning. This is because embedding vectors learned in the past are encoded by the parameters of the past, and do not reflect the update of the parameters made in the meantime.

7 and 8 are diagrams illustrating a conceptual diagram of a method for interpolating an embedding vector according to an embodiment of the present invention.

이고, 여러 차례 학습 사이클이 반복되어 현재

로 업데이트되었다. 과거 얼굴모델 파라미터

에 의해서 제 1 아이덴티티에 대한 제 1 내지 제 3 인스턴스가 제 a 내지 제 c 임베딩 벡터

(701-a 내지 701-c)로 인코딩되었고, 제 2 아이덴티티에 대한 제 4 내지 제 6 인스턴스가 제 d 내지 제 f 임베딩 벡터

(701-d ~ 701-f)로 인코딩되었다.Referring to FIG. 7 , the past face model parameters are

, and the learning cycle is repeated several times to

updated to Past face model parameters

The first to third instances of the first identity are the a to cth embedding vectors by

(701-a to 701-c), and the fourth to sixth instances for the second identity are the d to f th embedding vectors

(701-d to 701-f).

대신 업데이트된

를 이용하여 인코딩될 경우, 상기 제 1 아이덴티티에 대한 제 1 내지 제 3 인스턴스는 제 g 내지 제 i 임베딩 벡터

(701-g, 701-h, 701-i)로 인코딩되고, 상기 제 2 아이덴티티에 대한 제 4 내지 제 6 인스턴스는 제 j 내지 제 l 임베딩 벡터

(701-j, 701-k, 701-l)로 인코딩될 수 있다. 즉 파라미터의 업데이트(

)됨에 따라 인코딩된 결과 역시

로 달라지게 된다는 것이다. 결국 과거 학습 과정에서 사용된 임베딩 벡터

는, 오랜 학습이 반복된 이후에 그대로 사용할 경우 아래 수학식 6 만큼의 오차

를 발생시킬 수 있다는 의미이다.However, the parameters of the face model in the past

instead updated

When encoded using , the first to third instances of the first identity are

(701-g, 701-h, 701-i), and the fourth to sixth instances of the second identity are the j to first embedding vectors.

(701-j, 701-k, 701-l). i.e. update parameters (

), the encoded result is also

that will change to In the end, the embedding vectors used in the past learning process

is an error as much as Equation 6 below when used as it is after long learning is repeated.

This means that it can cause

는 그리 큰 값이 아닐 수 있다. 하지만 학습이 반복적으로 수행됨에 따라 오차

는 점진적으로 누적되어갈 것이다. 이러한 오차

를 최소화시키기 위하여 본 발명의 일실시예에서는 아래 수학식 7에 기초한 보간 함수

를 제안한다.If there are not many iterations of learning, the error

may not be very large. However, as the training is iteratively performed, the error

will gradually accumulate. these errors

In an embodiment of the present invention in order to minimize

suggest

는 보간되기 전 과거 임베딩 벡터이고,

는 보간된 과거 임베딩 벡터를 의미한다.here

is the past embedding vector before interpolation,

denotes an interpolated past embedding vector.

는 아래 수학식 8의 아이덴티티

에 대한 보간된 임베딩 벡터와 현재 임베딩 벡터 간의 평균 제곱 오차(expected squared error) 함수인

를 최소화 시켜야 한다. interpolation function

is the identity of Equation 8 below

is a function of the expected squared error between the interpolated embedding vector and the current embedding vector for

should be minimized

는 아이덴티티

에 대한 임베딩 벡터

의 기대값(expectation)을 의미한다.here

is the identity

embedding vector for

means the expected value of

에 대하여 편미분하면, 아래 수학식 9로 표현된다.Equation 8

By partial differentiation with respect to , it is expressed by Equation 9 below.

를 연산하면 아래 수학식 10과 같이 근사값을 도출할 수 있다.Optimal interpolation function using Equation 9

, an approximate value can be derived as shown in Equation 10 below.

는 제 2 대기열저장부(410-2)에 저장되는 과거 아이덴티티 대표 벡터로, 과거 학습 과정에서

가 제 1 대기열저장부(410-1)에 저장될 때 함께 저장된다.here

is a representative vector of the past identity stored in the second queue storage unit 410-2, and in the past learning process,

is stored together when stored in the first queue storage unit 410-1.

(701-a, 701-b, 701-c)의 과거 평균 지점(710, Class center)을 지시하는 제 1 중심 벡터와 현재 평균 지점(710')을 지시하는 제 2 중심 벡터의 차인

를 임베딩 벡터들

(701-a, 701-b, 701-c)에 더할 경우, 과거 평균 지점(710)과 현재 평균 지점(710')이 일치될 수 있을 것이다. 다만 아래 수학식 11에서와 같이 임베딩 벡터와 아이덴티티 벡터 간에 스케일을 맞추기 위한 상수

가 곱해질 수 있다. 이렇게 현재 평균 지점(710')으로 일치된 과거의 임베딩 벡터들을 보간된 임베딩 벡터

(701^*-1 내지 701^*-3)라 한다.As shown in Fig. 8, past embedding vectors for the first identity

The difference between the first center vector indicating the past average point 710 (Class center) of (701-a, 701-b, 701-c) and the second center vector indicating the current average point 710'

embedding vectors

When adding to (701-a, 701-b, 701-c), the past average point 710 and the current average point 710 ′ may coincide. However, as in Equation 11 below, a constant for adjusting the scale between the embedding vector and the identity vector

can be multiplied by An embedding vector interpolated by past embedding vectors matched with the current average point 710' in this way

(701 ^* -1 to 701 ^* -3).

는 보간되기 전 과거 임베딩 벡터이고,

는 보간된 후 과거 임베딩 벡터를 의미한다. 그리고

는 아이덴티티

에 대한 현재 아이덴티티 대표 벡터,

는 아이덴티티

에 대한 과거 아이덴티티 대표 벡터를 의미한다.here

is the past embedding vector before interpolation,

denotes the past embedding vector after interpolation. and

is the identity

current identity representative vector for

is the identity

It means a representative vector of the past identity for .

(502)를 보간 없이 이용하였다. 하지만, 제 2 대기열저장부(410-2)에 저장되는 과거 아이덴티티 대표 벡터

를 이용하여 보간(수학식 11을 이용)시킨 보간된 임베딩 벡터

를 대신 이용할 수 있을 것이다.Returning to FIG. 5 , in the above-described embodiment, the second embedding vector stored in the first queue storage unit 410 - 1 .

(502) was used without interpolation. However, the past identity representative vector stored in the second queue storage unit 410-2

An interpolated embedding vector interpolated using Equation 11 (using Equation 11)

could be used instead.

를 이용할 경우, 상술한 수학식 5의 제 2 손실 함수는 아래 수학식 12와 같이 바뀔 수 있을 것이다.Interpolated Embedding Vectors

In the case of using , the second loss function of Equation 5 may be changed to Equation 12 below.

)는 미니 배치뿐만 아니라 과거 학습된 데이터를 모두 고려하여 최적화될 수 있기 때문에, 반복으로 학습하더라도 정확도가 꾸준히 향상되는 것을 확인할 수 있다. 더군다나 본 발명의 실시예는, 기존 얼굴인식 장치에서 대기열저장부(410)만을 구비하는 것만으로도 손쉽게 적용할 수 있어, 기존 얼굴인식 장치들과 손쉽게 호환할 수 있다는 장점이 존재한다.As a result of applying the above-described methods, in the present invention, it is possible to consider a fairly large size of training data in one learning cycle. When the above-described methods are applied, the optimization of the model 110 was made based on the mini-batch input in the corresponding learning cycle, but the classifier 210 parameter (identity representative vector) was

) can be optimized by considering both the mini-batch as well as the data learned in the past, so it can be seen that the accuracy is steadily improved even if iteratively learning. Moreover, the embodiment of the present invention can be easily applied only by having only the queue storage unit 410 in the existing face recognition device, so there is an advantage of being easily compatible with the existing face recognition devices.

Hereinafter, the effects of the above-described embodiments of the present invention are demonstrated with experimental data.

9 is experimental data that proves the effect of interpolation according to an embodiment of the present invention.

9 (a) is an experimental result comparing the case with and without interpolation with respect to the cosine error between embedding vectors in the current learning cycle. Errors were calculated with randomly sampled instances over 64 learning cycles.

When the learning cycle repetition is small (32 or less), the cosine error value with and without interpolation is not significantly different. However, when many learning cycles are repeated (more than 32 times), the error increases, and it is determined that interpolation for past data is necessary. It will be apparent that the accuracy of the learning process may decrease if such an error increases.

and FIG. 9( b ) shows a Scatter Plot of past, present and interpolated embedding vector(s) for 64 instances of the past (64 learning cycles). Since the past embedding vector can be confirmed by accessing the current embedding vector after interpolation is applied, it can be confirmed that the proposed interpolation function is effective.

10 is a diagram comparing an increase in memory required as the amount of learning data increases.

The x-axis is the size of the entire training data (the size of the mini-batch/the storage size of the queue storage unit), and the y-axis is the axis of the memory usage.

As shown in the dotted line graph, if the size of the mini-batch is simply increased in a general way, when the size of the training data is to be increased, the size of the memory required linearly also increases. As shown in the solid line graph, if the size of the queue storage unit is increased according to the embodiment of the present invention, even if the training data is increased to about 8,192 instances, it can be confirmed that the required memory size of a single GPU does not exceed 32 GB. This is a relatively low number compared to about 952 GB of memory required to train 8,192 instances with a general method of increasing the size of a mini-batch.

* Experimental settings

는 L2-정규화되고 ArcFace에 의해 학습된다. 4 개의 동기화된 NVidia V100 GPU로 학습되었으며 각 GPU에 128 개 이미지의 미니 배치가 할당된다. 본 발명의 실시예(이하 BroadFace라고 호칭함)의 대기열저장부에는 각 GPU에 대해 64 개의 반복에 걸쳐 누적된 최대 8,192개의 임베딩 벡터가 저장되므로 대기열저장부의 총 크기는 4 개의 GPU에 대해 32,768개의 임베딩 벡터가 저장된다. 임베드 공간의 급격한 변경을 피하기 위해 BroadFace 네트워크는 softmax 기반 손실에 의해 학습된 사전 학습 네트워크에서 학습된다. SGD(stochastic gradient descent) 최적화(optimizer)를 채택했으며 학습 속도(learning rate)는 첫 번째 50k의 경우 5 X 10^-3, 20k의 경우 5 X 10^-4, 10k의 5 X 10^-5로, 무게 감소(weight decay)는 5 X 10^-4 그리고 운동량(momentum)은 0.9로 설정되었다.In the preprocessing process of this experiment, the facial image was normalized to 112 × 112 by transforming the facial region using five facial feature points (two eyes, nose, and mouth). As the backbone network, ResNet-100, which has been widely used recently, was used. After the res5c layer of ResNet-100, the batch normalization block, fully connected and batch normalization layers are applied to compute the 512-dimensional embedding vector. The encoded embedding vector and the weight vector of the linear classifier

is L2-normalized and trained by ArcFace. It was trained with 4 synchronized NVidia V100 GPUs and each GPU is assigned a mini-batch of 128 images. In the queue storage unit of the embodiment of the present invention (hereinafter referred to as BroadFace), a maximum of 8,192 embedding vectors accumulated over 64 iterations for each GPU are stored, so the total size of the queue storage unit is 32,768 embeddings for 4 GPUs. vector is stored. To avoid abrupt changes in the embedding space, the BroadFace network is trained on a pre-trained network trained by softmax-based loss. A stochastic gradient descent (SGD) optimizer is adopted, and the learning rate is 5 X 10 ^-3 for the first 50k, 5 X 10 ^-4 for 20k, 5 X 10 ^-5 for 10k, The weight decay was set to 5 X 10 ^-4 and the momentum was set to 0.9.

* training data set

In the experiment, a face recognition device was trained on MSCeleb-1M, which consisted of approximately 10 X 10 ⁶ instances (images) for 1 million identities. A modified version containing 3.8 X 10 ⁶ images for 85 X 10 ³ identities was used by removing the noise labels of the MSCeleb-1M. For the experiment, evaluation was performed on various data sets as follows.

- Labeled Faces in the Wild (LFW) is one of the most commonly used data sets for face identification. The dataset contains 13 X 10 ³ face images collected from the web for 5,749 different identities. LFW provides 6 X 10 ³ pairs of face images. Cross-Age LFW (CALFW) provides pairs with age changes, and Cross-Pose (CPLFW) provides pairs with varying poses in the LFW image.

- YTF (YouTube Faces) is a public data set consisting of videos downloaded from YouTube for face verification. The data set contains 3,425 videos for 1,595 different identities (people).

- MegaFace contains more than 1 X 10 ⁶ images with 690 X 10 ³ identities, and the recognition accuracy can be evaluated in the presence of various types of noise.

- CFP (Celebrities in Frontal-Prole) includes 500 classes (identities). Each subject has 10 frontal images and 4 profile images.

- AgeDB-30 contains 12,240 images with 440 identities with age changes, and is suitable for evaluating sensitivity to age changes.

- IARPA Janus Benchmark (IJB) is one of the most difficult data sets designed to evaluate unconstrained face recognition systems. IJB-B consists of 67 X 10 ³ face images, 7 X 10 ³ face videos, and 10 X 10 ³ non-face images. IJB-C consists of 138 X 10 ³ face images, 11 X 10 ³ facial animations, and 10 X 10 ³ non-face images adding new objects with increased occlusion and diversity of geographic origin to IJB-B.

LFW and YTF are widely used to evaluate verification performance in unrestricted environments. LFW containing image pairs evaluates the model by comparing the two embedding vectors for a given pair. YTF contains video, which is a set of images, from the shortest clip at 48 frames to the longest at 6,070 frames. To compare a pair of videos, YTF compares a pair of representative embedding vectors (identity representative vectors), which is the average embedding vector of images collected from each video. As can be seen in Table 1, the two data sets showed that other general methods also showed very high accuracy, but it can be seen that the embodiment of the present invention (BroadFace in the bottom row) outperforms other general methods.

CALFW, CPLFW, CFP-FP and AgeDB-30 are widely used to determine how effective they are against changes in posture or age. CALFW and AgeDB-30 have multiple instances of different age groups for a single identity, while CPLFW and CFP-FP have multiple instances of a single identity for different poses (front and profile). As can be seen in Table 2, compared with other general methods, the embodiment of the present invention (BroadFace in the bottom row) has high accuracy.

MegaFace is designed to evaluate face identification and verification tasks under conditions that contain a lot of noise. The present invention was evaluated with 'Megaface Challenge 1' with more than 50 million training data sets. As can be seen in Table 3, it was confirmed that the embodiment of the present invention (BroadFace) performed better than other face recognition models in both face identification and verification tasks. Also in the modified MegaFace (noisy labels removed), it is confirmed that the embodiment of the present invention (BroadFace) outperforms other face recognition models.

IJB-B and IJB-C are the most difficult training datasets to evaluate unconstrained face recognition. In IJB, since a pair of faces from an image or a moving picture is compared, evaluation is performed by constructing a feature embedded in a given image and a template feature representing the given moving image. Comparison of BroadFace and CosFace and BroadFace and ArcFace in verification work without enhancements such as horizontal flipping during evaluation. Looking at Table 4, it can be seen that the embodiment (BroadFace) of the present invention is significantly improved in all FAR standards. Compared with the results of ArcFace in IJB-B, BroadFace is superior to 8.25 percentage points at FAR = 1e ^-6 , 1.48 percentage points at FAR = 1e ^-5 , and 0.36 percentage points at FAR = 1e ^-4 .

Face recognition technology and image search technology have similarities in that they try to learn the optimal embedding space to compare a given pair of items, such as faces, clothes, and industrial products. Therefore, in order to prove the generalization ability of the method according to an embodiment of the present invention, the technique according to the embodiment of the present invention (BroadFace) is compared with a metric learning method recently proposed for image search technology. see.

* Experimental settings to apply to the image search area

ResNet-50, pre-trained in ILSVRC 2012-CLS, is used as the backbone network. ArcFace is used as the basic objective function and the queue size according to the embodiment of the present invention is set to 32 X 10 ³ . Other than that, the details of the learning process follow well-known methods. And follow standard evaluation protocol for evaluation. Evaluation was performed on two large data sets, In-Shop Clothes Retrieval (In-Shop) and Stanford Online Products (SOP), which have many classes similar to the face recognition data set. An embodiment of the present invention (BroadFace) can help a model to effectively learn a large number of classes from the two data sets.

In-Shop and SOP are standard training datasets for image retrieval techniques. The In-Shop has 11,735 classes of clothes. 25,882 images with 3,997 classes are used for training, and 26,830 images with the remaining 7,970 classes are split into query sets/gallery sets for evaluation. The SOP includes 22,634 industrial products. First, 59,551 images with 11,318 classes are used for training, and then 60,499 images with the remaining 11,316 classes are used for evaluation. Referring to Table 5, it can be seen that the basic model trained with ArcFace in this evaluation performed poorly compared to other modern methods. The embodiment of the present invention (BroadFace) significantly improves the recall of the basic model, and it can be seen that the performance is superior to that of other methods.

In the embodiment (BroadFace) of the present invention, there is only one queue size in which a hyper-parameter determines the maximum number of embedding vectors accumulated in past repeated learning. These parameters play a very important role in determining the recognition accuracy, and the use of a single parameter allows easy tuning of the assay. Referring to Table 6, it can be seen that the performance steadily increases as the queue size increases. It can be seen that without interpolation (without Compensation) the accuracy degradation occurs when the queue size is quite large. However, when interpolation is applied (with Compensation), the error of the past embedding vector is corrected to alleviate the performance degradation.

11 shows experimental data when the face recognition method according to an embodiment of the present invention is applied to an image search technology.

In the experiment shown, the change in recall was measured while changing the queue size from 0 to 32,000. It can be confirmed ( 1101 ) that the recall is continuously improved as the size of the queue increases by interpolation (with Compensation) according to an embodiment of the present invention. However, without the proposed interpolation (without Compensation), it was confirmed (1102) that the recall of the model was reduced when the queue size exceeded 16 X 10 ³ .

As confirmed above, according to the embodiment (BroadFace) of the present invention, it can be applied to all objective functions and various backbone networks as well as the face recognition area. As confirmed with reference to Table 4, it was confirmed that recognition accuracy was increased when the embodiment (BroadFace) of the present invention was applied to the objective functions of CosFace and ArcFace, which are widely used. In addition, the embodiment of the present invention (BroadFace) was applied to several backbone networks such as MobileFaceNet, ResNet-18 and ResNet-34. For light backbone networks such as MobileFaceNet and ResNet-18, the size of the embedding vector was set to 128, and for heavy backbone networks such as ResNet-34 and ResNet-100, it was applied by setting it to 512. Referring to Table 7, it can be confirmed that the embodiment of the present invention (BroadFace) is very effective for all backbone networks. In particular, ResNet-34 trained with the embodiment of the present invention (BroadFace) achieves similar performance to ResNet-100 trained only with ArcFace, even though ResNet-34 has much less GFlops.

An embodiment of the present invention (BroadFace) accelerates the learning process of face recognition technology and image search technology. In face recognition technology, many iterations are still needed to overcome small performance differences among various face recognition techniques on highly saturated data sets.

Referring to FIG. 11 ( b ), it is a graph in which acceleration of the learning process in image search is experimented to clearly show the effect of the embodiment (BroadFace) of the present invention. It can be seen that the embodiment of the present invention (BroadFace) achieves the highest performance much faster than other existing models. However, if there is no interpolation (without compensation), it can be seen that the performance deteriorates every time the learning is repeated.

Although the embodiments of the device for recognizing a face and the learning control method thereof have been described above according to the present invention, this will be described as at least one embodiment, and thereby the technical idea of the present invention and its configuration and operation are not limited. As not to be, the scope of the technical idea of the present invention is not limited / limited by the drawings or the description with reference to the drawings. In addition, the concepts and embodiments of the present invention presented in the present invention can be used by those of ordinary skill in the art as a basis for modifying or designing other structures in order to perform the same purpose of the present invention. , an equivalent structure modified or changed by those of ordinary skill in the art to which the present invention belongs is bound by the technical scope of the present invention described in the claims, and does not depart from the spirit or scope of the invention described in the claims Various changes, substitutions and changes are possible within the limits.

Claims (20)

In the learning control method of a device for recognizing a face,
receiving an image set for training;
calculating a first embedding vector for the image set based on a face recognition model;
first learning based on the calculated first embedding vector; and
A second learning step based on the calculated first embedding vector and second embedding vector,
The learning repeatedly performs the learning cycle of the calculating step to the second learning step,
The second embedding vector is characterized in that it is the first embedding vector calculated in a previously performed learning cycle,
A learning control method for a device that recognizes faces.
According to claim 1, wherein the first learning step,
Identity representative vector corresponding to a given person

calculating a proximity between an identity-representative vector and the first embedding vector; and
Comprising the step of adjusting the parameter to increase the calculated proximity,
The adjusted parameter is a face recognition parameter used in the face recognition model.

characterized in that
A learning control method for a device that recognizes faces.
According to claim 1, wherein the second learning step,
Identity representative vector corresponding to a given person

, determining a proximity between the first embedding vector and the second embedding vector; and
The identity representative vector to increase the determined proximity

characterized in that it comprises the step of adjusting
A learning control method for a device that recognizes faces.
The method of claim 3, wherein the second learning step comprises:
Further comprising the step of compensating the second embedding vector,
Determining the proximity is characterized in that the determination based on the interpolated second embedding vector,
A learning control method for a device that recognizes faces.
5. The method of claim 4, wherein the interpolating comprises:
specific person

A representative vector of the identity of the learning cycle currently being performed on

and the identity representative vector of the previously performed learning cycle.

calculating the difference between
Characterized in that interpolation based on the calculated difference,
A learning control method for a device that recognizes faces.
The method of claim 5, wherein the interpolating comprises:
Further comprising the step of adjusting the scale to the calculated difference,
Interpolating based on the scaled difference, characterized in that
A learning control method for a device that recognizes faces.
4. The method according to any one of claims 2 and 3, wherein the determining of the proximity is a loss function defined by the following equation:

Calculation is done by

in the formula

ego,

is the set of images,

is the number of image sets,

is the first embedding vector,

is a labeled person,

is the L2 normalized vector,

Is

Characterized in that it is an identity representative vector for
A learning control method for a device that recognizes faces.
The method of claim 1,
The image set includes a plurality of images of a plurality of persons,
The first embedding vector is characterized in that it is calculated for each of the plurality of images,
A learning control method for a device that recognizes faces.
9. The method of claim 8,
identity representative vector

characterized in that it further comprises the step of storing the corresponding to each of the plurality of persons,
A learning control method for a device that recognizes faces.
The method of claim 1,
The method further comprising: updating the second embedding vector so that the calculated first embedding vector is included in the second embedding vector;
The updated second embedding vector is characterized in that it is used in a learning cycle performed after the update,
A learning control method for a device that recognizes faces.
A device for recognizing a face, comprising:
a face recognition model that receives an image set for training and calculates a first embedding vector for the input image set;
a first learning unit performing first learning based on the calculated first embedding vector; and
Comprising a second learning unit to learn a second based on the calculated first embedding vector and the second embedding vector,
The learning repeatedly performs the learning cycle of the calculation, the first and the second learning,
The second embedding vector is characterized in that it is the first embedding vector calculated in a previously performed learning cycle,
Face recognition device.
The method of claim 11, wherein the first learning unit,
Identity representative vector corresponding to a given person

Calculate the proximity between (identity-representative vector) and the first embedding vector,
Adjust the parameters to increase the calculated proximity,
The adjusted parameter is a face recognition parameter used in the face recognition model.

characterized in that
Face recognition device.
The method of claim 11, wherein the second learning unit,
Identity representative vector corresponding to a given person

, determine the proximity between the first embedding vector and the second embedding vector,
The identity representative vector to increase the determined proximity

characterized by adjusting
Face recognition device.
The method of claim 13, wherein the second learning unit determines the proximity,
interpolating the second embedding vector;
Characterized in determining based on the interpolated second embedding vector,
Face recognition device.
The method according to claim 14, wherein the second learning unit performs the interpolation,
specific person

A representative vector of the identity of the learning cycle currently being performed on

and the identity representative vector of the previously performed learning cycle.

Calculate the difference between
characterized by interpolating based on the calculated difference,
Face recognition device.
The method according to claim 15, wherein the second learning unit performs the interpolation,
Adjust the scale to the calculated difference,
Interpolating based on the scaled difference, characterized in that
Face recognition device.
[Claim 14] The loss function according to any one of claims 12 and 13, wherein the first and second learning units determine proximity in a loss function defined by the following equation.

Calculation is done by

in the formula

ego,

is the image set,

is the number of image sets,

is the first embedding vector,

is a labeled person,

is the L2 normalized vector,

Is

Characterized in that it is an identity representative vector for
Face recognition device.
12. The method of claim 11,
The image set includes a plurality of images of a plurality of persons,
The first embedding vector is characterized in that it is calculated for each of the plurality of images,
Face recognition device.
19. The method of claim 18,
identity representative vector

Characterized in that it further comprises a classification unit for storing the corresponding to each of the plurality of people,
Face recognition device.
12. The method of claim 11,
Further comprising a queue storage unit for updating the second embedding vector so that the calculated first embedding vector is included in the second embedding vector,
The updated second embedding vector is characterized in that it is used in a learning cycle performed after the update,
Face recognition device.

Images