KR20210114308A - Apparatus and method for recognizing heterogeneous faces based on relationship between component - Google Patents

Abstract

The present invention provides an apparatus and a method for recognizing different faces. The apparatus for recognizing different faces comprises: an element feature extraction unit dividing a face image acquired regardless of a domain representing an acquired method into a plurality of areas in accordance with a pre-learned pattern estimation method, and estimating a pattern of each divided area to acquire a feature map consisting of a plurality of feature vectors; a vector position and distance calculation unit calculating a position and a relative distance of each of the plurality of feature vectors in the feature map to couple the position and the relative distance to the corresponding feature vector; a vector relationship modeling unit considering the plurality of feature vectors to which the positions and the relative distances are coupled as a plurality of node vectors in a virtual common space, and acquiring a relationship graph model in which relationships between the plurality of node vectors are expressed as edges; a relationship model update unit acquiring weights in accordance with semantic importance of the plurality of node vectors of the relationship graph model in accordance with a pre-learned pattern estimation method, and applying the acquired weights to the plurality of node vectors to acquire a correction relationship graph model; and a face recognition unit similarity-comparing the correction relationship graph model with a plurality of reference relationship graph model which are correction relationship graphs previously stored by mapping subjects to recognize a subject of an acquired face image. A face can be accurately identified from various face images, and the apparatus for recognizing different faces can be efficiently trained with a small amount of training data.

Description

Apparatus and method for heterogeneous face recognition based on extracting relationships between elements

The present invention relates to an apparatus, method and method for recognizing heterogeneous faces, and to an apparatus and method for recognizing heterogeneous faces based on extracting relationships between elements.

The field of face recognition has developed radically with the advent of deep learning technology. In the past, face recognition using deep learning was mainly performed through matching between images acquired in the same way using the same type of sensor. For example, in the past, a face was recognized by matching a face between color images.

However, there are not only color images, but also images acquired using infrared (NIR) or thermal infrared (TIR) sensors in poor lighting conditions. There are images obtained in this way. And each image may contain more useful information than a color image depending on the situation. For example, heterogeneous face recognition technology acquired in different ways, such as infrared-to-vis or sketch-to-photo, is applied to real life such as CCTV images taken at night and montage face recognition of criminals. can be very useful in

Accordingly, recently, research into the field of heterogeneous face recognition capable of recognizing a face even between different types of sensors and images obtained by various methods is being actively conducted. However, heterogeneous face recognition has a large difference between images according to the acquisition method, and there is a limitation in that there is not enough heterogeneous face dataset to sufficiently train the network even if deep learning technology is applied.

Korean Patent Publication No. 10-2017-0140519 (published on December 21, 2017)

SUMMARY OF THE INVENTION It is an object of the present invention to provide a heterogeneous face recognition apparatus and method capable of accurately identifying a face from various face images obtained in different ways.

Another object of the present invention is to provide a heterogeneous face recognition apparatus and method that can be efficiently learned with little learning data by generating a relationship graph model in a virtual common space using relationships between components.

A heterogeneous face recognition apparatus according to an embodiment of the present invention for achieving the above object divides an acquired face image irrespective of a domain indicating an acquired method into a plurality of regions according to a pre-learned pattern estimation method, and divides the an element feature extraction unit for estimating the pattern of each region to obtain a feature map composed of a plurality of feature vectors; a vector position and distance calculator for calculating a position and a relative distance of each of the plurality of feature vectors in the feature map and combining them with a corresponding feature vector; a vector relationship modeling unit that considers each of a plurality of feature vectors in which a position and a relative distance are combined as a plurality of node vectors on a virtual common space, and obtains a relationship graph model in which the relationship between the plurality of node vectors is expressed as an edge; A relation in which a weight according to the semantic importance of each of the plurality of node vectors of the relationship graph model is obtained according to a pre-learned pattern estimation method, and a correction relationship graph model is obtained by weighting the obtained weights to each of the plurality of node vectors model update unit; and a face recognition unit for recognizing the subject of the obtained face image by comparing the correction relation graph model with a plurality of reference relation graph models that are pre-stored correction relation model graphs to which the subject is mapped.

The vector relationship modeling unit projects each of a plurality of feature vectors in which a position and a relative distance are combined as a plurality of node vectors on a virtual common space, and represents a plurality of relationships between the plurality of node vectors according to the plurality of node vectors. The relation graph model may be generated by setting the edge of , based on the relation weight obtained by learning.

The vector relationship modeling unit propagates the relationship to the node vector connected to each edge according to the set plurality of edges, updates the plurality of node vectors to the next state, and updates the edge again according to the state of the updated node vector. It can be repeated a specified number of times.

The relation model update unit may include: a representative value calculation unit which obtains a representative value of each of the plurality of node vectors in a predetermined manner to obtain a representative graph model expressed as a representative value; an encoder that encodes the representative graph model according to a pre-learned pattern estimation method to obtain an importance vector indicating the importance of each of a plurality of node vectors of the relation graph model; a decoder that receives the importance vector and decodes it according to a pre-learned pattern recovery method to obtain a weight relationship model graph having a weight according to the importance of each node vector of the relationship graph model as a node; and an update unit configured to obtain the correction relationship model graph by weighting a weight of a corresponding node of the weight relationship model graph to each of the plurality of node vectors of the relationship graph model.

The heterogeneous face recognition apparatus further includes a learning unit for learning the vector relation modeling unit and the relation model updating unit, wherein the learning unit includes an anchor that is a face image of a subject as a reference for a face image applied as training data, and the anchor; A correction relationship model graph is obtained for each positive, which is a face image of a different domain for the same subject, and a negative, which is a face image for a different subject, with an anchor, and between the anchor and positive, respectively, corresponding multiple node vectors on the common space. Calculate the distance and the distance between the anchor and the corresponding plurality of node vectors of each negative based on the similarity, and calculate the distance between the calculated anchor and the corresponding plurality of node vectors of each of the positive and negative anchors and the corresponding plurality of each of the anchors and negatives. We can calculate and backpropagate the loss based on the ratio between the distances between the node vectors.

The learning unit is a predetermined margin ( _{s i} ^p ) between an anchor and a distance (s i p ) between an anchor and a corresponding plurality of node vectors of each positive and a distance (s _i ^{n ) between the anchor and a corresponding plurality of node vectors of each negative (s i n )} m) by additionally applying the equation

(where [] ₊ is a positive function that takes only positive values), we can calculate _{the loss (L tripletconditional ).}

The heterogeneous face recognition apparatus further includes a facial feature database in which the plurality of reference relationship graph models are mapped to an identifier of a subject and stored, and the face recognition unit has a reference relationship having a similarity greater than or equal to a predetermined reference similarity to the correction relationship graph model. When the graph model is searched, an identifier mapped to the searched reference relationship graph model may be output as a face recognition result.

The heterogeneous face recognition apparatus may recognize a subject by receiving one of a color image, an infrared image, a thermal infrared image, or a sketch image according to a domain.

A heterogeneous face recognition apparatus and method according to another embodiment of the present invention for achieving the above other object divides an acquired face image irrespective of a domain indicating an acquired method into a plurality of regions according to a pre-learned pattern estimation method, and , obtaining a feature map composed of a plurality of feature vectors by estimating the pattern of each divided region; calculating a position and a relative distance of each of the plurality of feature vectors in the feature map and combining them with a corresponding feature vector; obtaining a relationship graph model in which each of a plurality of feature vectors combined with a position and a relative distance is regarded as a plurality of node vectors on a virtual common space, and relationships between the plurality of node vectors are expressed as edges; Obtaining a weight according to the semantic importance of each of the plurality of node vectors of the relationship graph model according to a pre-learned pattern estimation method, and obtaining a corrected relationship graph model by weighting the obtained weights on each of the plurality of node vectors ; and recognizing the subject of the obtained face image by comparing the correction relation graph model with a plurality of reference relation graph models that are pre-stored correction relation model graphs to which the subject is mapped.

Therefore, the heterogeneous face recognition apparatus and method according to an embodiment of the present invention generates a relationship graph model in a virtual common space using distance relationships between components, and compares the node vector similarity of the generated relationship graph model In addition to being able to accurately identify a face from various face images obtained in this way, it can be efficiently learned with a small amount of training data.

1 shows a schematic structure of a heterogeneous face recognition apparatus according to an embodiment of the present invention.
FIG. 2 is a diagram for explaining an operation of the element feature extracting unit of FIG. 1 .
FIG. 3 is a diagram for explaining the operation of the vector position and distance calculator of FIG. 1 .
4 and 5 are diagrams for explaining an operation of the vector relationship modeling unit of FIG. 1 generating a relationship graph model between feature vectors and propagating the relationship.
6 shows a detailed configuration of the relation model update unit of FIG. 1 .
FIG. 7 is a diagram for explaining a concept in which the relationship model update unit of FIG. 6 updates the relationship graph model.
8 and 9 are diagrams for explaining a concept in which a learning unit learns a heterogeneous face recognition apparatus.
10 shows a change in a feature vector projected to a common space according to whether or not a margin-applied learning is performed.
11 illustrates a heterogeneous face recognition method according to an embodiment of the present invention.
12 is a diagram illustrating in detail a step of updating the relationship graph model of FIG. 11 .

In order to fully understand the present invention, the operational advantages of the present invention, and the objects achieved by the practice of the present invention, reference should be made to the accompanying drawings illustrating preferred embodiments of the present invention and the contents described in the accompanying drawings.

Hereinafter, the present invention will be described in detail by describing preferred embodiments of the present invention with reference to the accompanying drawings. However, the present invention may be embodied in various different forms, and is not limited to the described embodiments. In addition, in order to clearly describe the present invention, parts irrelevant to the description are omitted, and the same reference numerals in the drawings indicate the same members.

Throughout the specification, when a part "includes" a certain component, it means that other components may be further included, rather than excluding other components unless otherwise stated. In addition, terms such as "... unit", "... group", "module", and "block" described in the specification mean a unit that processes at least one function or operation, which is hardware, software, or hardware. and a combination of software.

1 shows a schematic structure of a heterogeneous face recognition apparatus according to an embodiment of the present invention.

Referring to FIG. 1 , the heterogeneous face recognition apparatus according to the present embodiment includes a heterogeneous image acquisition unit 100 , an element feature extraction unit 200 , a vector position and distance calculation unit 300 , a vector relationship modeling unit 400 , It may include a relation model update unit 500 , a face recognition unit 600 , and a facial feature database 700 .

The heterogeneous image acquisition unit 100 acquires heterogeneous face images obtained in various ways. Here, the face image may be a heterogeneous face image obtained from different types of sensors such as a color image sensor or infrared and thermal infrared sensors, or may be an image obtained without using a sensor, such as a sketch image of a face.

Here, the heterogeneous image acquisition unit 100 does not need to distinguish a method in which each face image is acquired. However, the heterogeneous image acquisition unit 100 may acquire each face image by dividing it into domains according to the acquired method. That is, the heterogeneous image acquisition unit 100 may acquire a plurality of face images by dividing them into a color domain, an infrared domain, a thermal infrared domain, and a sketch domain according to an image acquisition method.

However, the heterogeneous image acquisition unit 100 classifies and acquires face images obtained by different methods into different domains according to the manner in which the respective face images are obtained when learning the heterogeneous face recognition apparatus. That is, the color domain image, the infrared domain image, the thermal infrared domain image, and the sketch domain image may be separately acquired and stored.

In addition, the heterogeneous image acquisition unit 100 may acquire and store identifiers by mapping the identifiers according to subjects that are subjects of a facial image obtained by being divided by domains during training of the heterogeneous face recognition apparatus. That is, if each face image obtained from the same or different domains is a face image of the same subject, the same identifier is assigned, and if the face image is of a different subject, different identifiers are assigned and stored.

However, domain division and identifier mapping of the face image are for learning of the heterogeneous face recognition apparatus, and when the heterogeneous face recognition apparatus according to the present embodiment is used for actual face recognition, the heterogeneous image acquisition unit 100 determines the obtained face It does not distinguish domains of images and does not map identifiers. This is because the heterogeneous face recognition apparatus according to the present embodiment is trained in advance to recognize a face regardless of the domain of the face image.

The element feature extracting unit 200 obtains a plurality of feature vectors by extracting each element feature according to a pattern estimation method learned in advance for each of the face images obtained by the heterogeneous image obtaining unit 100 .

FIG. 2 is a diagram for explaining an operation of the element feature extracting unit of FIG. 1 .

As shown in FIG. 2 , when a face image is applied, the element feature extraction unit 200 extracts a feature for each element of the face image to obtain a feature map. The element feature extractor 200 may divide the face image into a plurality of regions in a predetermined manner, and may obtain a feature map by estimating a pattern of each divided region. The feature map may be composed of a plurality of feature vectors whose patterns are estimated corresponding to each region, and each of the plurality of feature vectors may be obtained as a one-dimensional vector in the Z-axis direction as shown in FIG. 2 , for example.

The element feature extracting unit 200 that extracts elemental features from a face image to obtain a feature map may be implemented as an artificial neural network, and may be learned according to a loss propagated during learning of a heterogeneous face recognition device, but from the image. Since the artificial neural network for extracting features is publicly available, an artificial neural network that has already been trained may be used.

The vector position and distance calculator 300 calculates a position of each of a plurality of feature vectors constituting the feature map, and calculates a relative distance between the plurality of feature vectors based on the calculated positions of the feature vectors.

Each of the feature vectors extracted by the element feature extraction unit 200 extracts a feature of a corresponding region in the face image, which can be seen as extracting features for each component from the subject's face. And the relative distance between these components can be viewed as an independent feature independent of the domain, which is a method of acquiring a face image with a unique characteristic of each subject's face. For example, the distance between the human eye and the eye and the distance between the mouth and the chin may be viewed as unique characteristics of each subject.

Accordingly, in the present embodiment, the vector position and distance calculator 300 calculates the position and the relative distance of each feature vector in the feature map, and combines the calculated position and the relative distance with the corresponding feature vector.

FIG. 3 is a diagram for explaining the operation of the vector position and distance calculator of FIG. 1 .

As shown in FIG. 3 , the vector position and distance calculator 300 calculates the x coordinate (x _i ) and the y coordinate (y _j ), which are absolute position information, for each of a plurality of feature vectors of the feature map, and calculates Based on the position coordinates (x _i , y _{j ) of each of the plurality of feature vectors, the relative distance (r i,j} ) of each feature vector with respect to _{the center (x 0} , y ₀ ) of the feature map according to Equation 1 ) is calculated and obtained.

Here, it has been described that the vector position and distance calculator 300 calculates the position coordinates (x _i , y _j ) and the relative distances (r _i,j ) of each feature vector on the XY plane, but this means that the feature vector of the feature map is This is because it is assumed that it is a one-dimensional vector in the Z-axis direction, and the plane for calculating the position coordinates and the relative distance may be changed according to the direction constituting the feature vector.

In addition, the vector position and distance calculator 300 calculates the position coordinates (x _i , y _j ) and the relative distances (r _i,j ) of each of the plurality of feature vectors, the calculated position coordinates (x _i , y _j ) and the relative distance (r _i,j ) are concatenated into the corresponding feature vector. _{As an example, the calculated position coordinates (x i} , y _j ) and relative distances (r _i,j ) are further added in the Z-axis direction to each of the plurality of feature vectors of the feature map.

_{When the position coordinates (x i} , y _j ) and the relative distance (r _i,j ) are combined in each of the plurality of feature vectors, the vector relationship modeling unit 400 sets the position coordinates (x _i , y _j ) and the relative distance (r) Each of a plurality of feature vectors combined with _{i,j ) is set as a node vector (n i} ), and the relationship between the set node vectors is set as an edge (e _i,j ) to establish the relationship between the feature vectors in a virtual common space. Create a relationship graph model that is represented as a graph on the top.

4 and 5 are diagrams for explaining an operation of the vector relationship modeling unit of FIG. 1 generating a relationship graph model between feature vectors and propagating the relationship.

The vector relationship modeling unit 400 first projects each of a plurality of feature vectors in a virtual common space to set a corresponding node vector (n _i ), as shown in FIG. 4 . As described above, since the position and distance of the feature vectors represent the unique characteristics of the subject's face regardless of the domain, a plurality of _{location coordinates (x i} , y _j ) and relative distance (r _{i,j )} Each of the feature vectors may be expressed as _{a node vector n i} on a common space that is not limited to a domain.

_{And a plurality of edges (e i,j} ^t ) connected between each set node vector (n _i ^t ) indicating a relationship between the node vectors are set. ^{In FIG. 4 , t indicated} in each node vector (n _i t ) and edge (e _i,j ^t ) indicates a current state. That is, n _i ^t and e _i,j ^t mean the edge from the i-th node vector and the i-th node vector (n _i ^t ) to the j-th node vector (n _j ^t _{) in the current state, and n i} ^t+1 and e _i,j ^t+1 means the edge from the i-th node vector and the i-th node vector (n _i ^t ) to the j-th node vector (n _j ^t ) in the next state.

When the total number of feature vectors in the feature map is D, the vector relationship modeling unit 400 generates D node vectors (N = {n ₁ ^t , n ₂ ^t , …, n _D ^t }) in the graph space, and , a plurality of edges (e _i,j ^t ) connecting between the generated D node vectors may be set according to Equation (2).

Here, σ is an activation function, and may be, for example, a sigmoid. And W _e is updated by learning as a relation weight, and T denotes a transpose matrix.

_{And the vector relationship modeling unit 400 calculates the next state (n i} ^t+1 ) of the plurality of node vectors connected to _{the edge (e i,j} ^t ) of the current state calculated according to Equation 2 according to Equation 3 to update

As described above, since the plurality of node vectors (n _i ) and the plurality of edges (e _i,j ) have a mutual relationship, the plurality of node vectors (n _i ) and the plurality of edges (e _i,j ) are alternately The state can be updated repeatedly. This update process is called relationship propagation, and relationship propagation may be repeatedly performed a predetermined number of times.

The relation model update unit 500 receives the relation graph model modeled by the vector relation modeling unit 400 and the relation propagated relation graph model is authorized, and a semantically more valuable node among _{a plurality of node vectors (n i ) of the relation graph model applied.} vector each node vector (n _i) element weight (w _i) in advance corresponding to acquisition according to the learning pattern estimation method between each node of the graph model vectors corresponding to such that the higher attention to the (n _i) ( We update the relation graph model by weighting n _{i ).}

FIG. 6 shows a detailed configuration of the relationship model updater of FIG. 1 , and FIG. 7 is a diagram for explaining the concept of the relationship model updater updating the relationship graph model of FIG. 6 .

Referring to FIG. 6 , the relationship model updater 500 may include a representative value calculator 510 , an encoder 520 , a decoder 530 , and an updater 540 .

The representative value calculation unit 510 receives an updated relation graph model generated and relation propagated by the vector relation modeling unit 400 as shown in FIG. the calculating a representative value (z _i) of the updated nodes vector (n _i ^{t + 1),} represented by the representative values (z _i) of the node vector (n _i ^{t + 1)} calculated as in (b) represented in accordance with Acquire a graph model.

_{The representative value calculator 510 may calculate the representative value z i} of the average value of the node vector in various ways, but here, as an example, the average value of the node vector is calculated as the representative value z _{i according to Equation 4} Assume

Here, C represents the length of the node vector.

The encoder 520 encodes the representative graph model according to the pre-learned pattern estimation method, and obtains an importance vector indicating the importance _{of each node vector (n i} ^{t+1 ) of the relation graph model as shown in (c).}

And the decoder 530 receives the importance vector obtained from the encoder 520 and decodes it according to the pattern recovery method learned in advance, and as shown in (d), the importance _{of each node vector (n i} ^{t+1) of the relationship graph model.} A weight relation model graph having a weight (w _{i ) according to}

The update unit 540 receives the relation graph model and the weight relation model graph, and adds _{the weight w i} of the corresponding node of the weight relation model graph to each of _{a plurality of node vectors (n i} ^t+1 ) of the relation graph model. By updating the relation model graph by weighting, a correction relation model graph is obtained as shown in (e).

The face recognition unit 600 receives the correction relation model graph from the relation model update unit 500 and is based on each of a plurality of node vectors (w _i n _i ) of the applied correction relation model graph and the facial feature database 700 . By comparing the degree of similarity between corresponding node vectors of the plurality of stored reference relationship graph models, a reference relationship graph model that is equal to or greater than a predetermined reference relationship graph model is searched for. And when the reference relationship graph model that is equal to or greater than the reference similarity is searched for, it is determined that the target of the obtained relationship graph model and the target of the reference relationship graph model are the same person. Accordingly, the face recognition unit 600 obtains and outputs the identifier mapped to the searched reference relationship graph model from the facial feature database 700 .

_{That is, the face recognition unit 600 includes a plurality of node vectors (w i} n _i ) and facial features of the correction relationship graph model generated from the facial images obtained and applied in various ways regardless of the domain to the heterogeneous image acquisition unit 100 . According to the degree of similarity between the plurality of node vectors of each of the plurality of reference relationship graph models stored in advance in the database 700 , the subject of the facial image applied to the heterogeneous image acquisition unit 100 is stored in the facial feature database 700 as the reference relationship graph model Face recognition is performed by determining whether or not the subject is a pre-stored subject, and an identifier of the subject, which is the subject of the face image, is output as a result of face recognition.

The facial feature database 700 maps and stores the correction relationship graph model previously obtained from the face image as a reference relationship graph model with the identifier of the subject who is the subject of the face image, and stores the correction relationship graph at the request of the face recognition unit 600 . The model is transmitted to the face recognition unit 600 .

As the facial feature database 700 pre-stores a plurality of reference relationship graph models mapped to identifiers, when the face recognition unit 600 searches for a reference relationship graph model having a degree of similarity or higher, the reference relationship graph model corresponding to the searched reference relationship graph model is stored in advance. You can pass an identifier.

As described above, when face images of various domains are applied, the heterogeneous face recognition apparatus according to this embodiment obtains a feature map by extracting component elements, that is, feature vectors for each area of the face image from the applied face image, and obtains After calculating and combining the positions and relative distances of each of the multiple feature vectors in the feature map, project them to a relation graph model on a virtual common space and correct the node vectors according to their importance. Face recognition is performed by comparing the unique features of each face image with each other regardless of the domain of the image. Therefore, even when face images of various different domains are applied, face recognition can be accurately performed.

However, in order for the above-described heterogeneous face recognition apparatus to perform face recognition, it must be learned in advance. In particular, the vector relation modeling unit 400 and the relation model update unit 500 implemented as an artificial neural network should be learned. Accordingly, the heterogeneous face recognition apparatus according to the present embodiment may further include a learning unit 800 .

8 and 9 are diagrams for explaining a concept in which a learning unit learns a heterogeneous face recognition apparatus.

The learning unit 800 according to this embodiment calculates the loss based on the difference between the _{node vectors (w i} n _i ) of the correction relationship model graph obtained from the face images obtained for the same subject and different subjects regardless of the domain. The vector relation modeling unit 400 and the relation model updating unit 500 may be trained by calculating and backpropagating the calculated loss.

Hereinafter, regardless of the domain of the face image, the face image of the subject, which is the basis of the face image, is referred to as an anchor, the face image of the subject same as the anchor is referred to as positive, and the face of the subject different from the anchor is referred to as an anchor. The image is called negative.

In (a) of FIG. 8, the left side is an infrared domain image of the anchor, the middle face image is a color domain image, but it is positive because it is a face image for the same subject, and (c) is an infrared domain image similar to (a), but different from the anchor. Since it is a face image of the subject, it is negative.

In this embodiment, the learning unit 800 calculates a loss using a triplet learning method that compares positive and negative with the anchor. As shown in Fig. 8 (b), the triplet learning method compares the distance (similarity) between the _{anchor and a number of node vectors (w i} n _i ) of the correction relationship model graph in which the positive and negative are projected on a virtual common space. The positive node vector is closer to the corresponding node vector of the anchor, while the positive node vector is a method of performing learning by calculating the loss so that it is further away from the corresponding node vector of the anchor.

Assuming that the node vector of the anchor is x _i ^a , the positive node vector is x _i ^p , and the negative node vector is x _i ⁿ , the anchor node vector ( x _i ^a ) and the positive node are in the common space. vector can be calculated as (x _i ^p) the distance (s ^p _i) is the cosine similarity (cosine similarity) using a function ^{_{(CS) s i p = CS}} (x i a, x i p) between. And the distance (s _i ⁿ ) between the node vector of the anchor (x _i ^a ) and the node vector of the negative (x _i ⁿ ) is also used using the cosine similarity function (CS) to s _i ⁿ = CS(x _i ^a , x _{i )} ⁿ ) can be calculated.

The anchor and the positive node to the example vector _{^{(x i a, x i p}} ) , while the distance (s _i ^p) between the computed to be 0.2 as shown in Figure 9 (a), an anchor and a negative node vector (x _i ^a, x The distance (s _i ⁿ _{) between i} ⁿ ) may be calculated as 0.6.

_{And the learning unit 800 calculates the distance (s i} ^p ) between the calculated anchor and the positive node vector (x _i ^a , x _i ^p ) and the distance between the anchor and the negative node vector (x _i ^a , x _i ⁿ ) ( s _i ⁿ is calculated by the loss according to the ratio of the back propagation), it is possible to learn the relationship vector modeling unit 400 and the relationship model updating section 500. the

However, simply the ratio of the distance (s _i ⁿ⁾ between an anchor and the positive node vector (x _i ^a, x _i ^p) the distance between the (s _i ^p) and the anchor and a negative node vector (x _i ^a, x _i ⁿ⁾ In the case of backpropagation by calculating the loss, the boundary that separates the positive and the negative based on the distance from the anchor in the common space after learning appears linearly, resulting in a very sensitive change in the face recognition result. That is, even a small change in the applied face image may cause a positive and negative change to be discriminated.

The learning unit 800 according to this embodiment is between the anchor and the positive node vector (x _i ^a, x _i ^p), the distance (s _i ^p) and the anchor and a negative node vector (x _i ^a, x _i ⁿ⁾ between _{A loss (L tripletconditional} ) is calculated according to Equation (5) by additionally applying a margin (m) to the ratio of the distance (s _i ^{n ) of .}

where [] ₊ is a positive function that takes only positive values.

In this way, when the learning unit 800 additionally applies the margin m, as shown in FIG. 9B , a margin region is generated between the positive region and the negative region in the common space according to the size of the margin. This makes it possible to clearly distinguish between positives and negatives, which can slow down the sensitivity of facial recognition. That is, robust face recognition can be performed.

10 shows a change in a feature vector projected to a common space according to whether or not a margin-applied learning is performed.

In FIG. 10 (a), when the learning unit 800 performs learning without applying the margin m, positive and negative node vectors (x _i ^a , x _i ^p , x with an anchor projected to a common space) _i ⁿ ), and (b) shows the anchor and positive and negative node vectors (x _i ^a , x _i ^p , x _i ⁿ ) projected to the common space when learning is performed by applying the margin (m). indicates.

Comparing (a) and (b), compared to the case where the margin (m) is not applied, when learning is performed by applying the margin (m), between the anchor and the positive node vector (x _i ^a , x _i ^p ) the distance (s ^p _i) is the distance (s _i ⁿ⁾ between contrast makin closer to, an anchor node and a negative vector (x _i ^a, x _i ⁿ⁾ it can be seen further more apart. That is, it can be seen that the face recognition apparatus is more efficiently learned to accurately perform face recognition.

11 is a diagram illustrating a heterogeneous face recognition method according to an embodiment of the present invention, and FIG. 12 is a diagram illustrating in detail an update step of the relation graph model of FIG. 11 .

11 and 12 , the heterogeneous face recognition method according to the present embodiment first acquires face images obtained in various ways regardless of a domain ( S10 ). Then, the obtained face image is divided into a plurality of regions in a predetermined manner, a pattern of each divided region is estimated, and a feature vector is extracted for each element to obtain a feature map (S20).

When the feature map is obtained, the position and relative distance of each of the plurality of feature vectors of the feature map are calculated (S30). Then, the calculated positions and relative distances for each of the plurality of feature vectors are combined with the corresponding feature vectors (S40).

Thereafter, a plurality of feature vectors of the feature map in which positions and relative distances are combined are regarded as a plurality of node vectors on a virtual common space, and a relationship graph model in which the relationships between the plurality of node vectors are expressed as edges (S50) . In the relation graph model acquisition step (S50), each of a plurality of feature vectors of the feature map is projected into a common space and set as a corresponding node vector (n _i ) to generate a relation graph model (S51). And by connecting a plurality of set node vectors (n _i ^t ) to each other, a plurality of edges (e _i,j ^t ) representing the relationship between the plurality of node vectors (n _i ^t ) are set according to Equation 2, and the set plurality By propagating the relationship based on the edge (e _i,j ^t ) of , the next state of a plurality of connected node vectors (n _i ^t+1 ) is calculated and propagated according to Equation 3 (S52). Then, it is determined whether or not to end the relationship propagation (S53). If it is determined that the relationship propagation does not end, the node vector (n _i ^t+1 ) in the next state is propagated again to the multiple edges (e _i,j ^t+1 ), and the relationship propagated multiple edges (e _{i ,j} ^t+1 ) to repeatedly perform relationship propagation to obtain the node vector (n _i ^{t+2 ) of the next state again (S52).}

When the relationship propagation is performed a predetermined number of times to terminate the relationship propagation, according to the pre-learned pattern estimation method to focus attention _{on the semantically important node vector (n i} ^{t+1 ) in the relationship graph model obtained by the relationship propagation} The element weight w _i is obtained, and the relation graph model is updated by weighting the _{obtained element weight w i} on each node vector n _i ^{t+1 of the relation graph model ( S60 ).}

Referring to FIG. 12 , the relation graph model update step ( S60 ) first _{calculates representative values of a plurality of node vectors (n i} ^t+1 ) of the relation graph model in a predetermined manner, and a representative composed of the calculated representative values. A graph model is obtained (S61). The representative value of the node vector (n _i ^{t + 1)} may be obtained as an average value vector of the node (n _i ^{t + 1)} elements as an example.

When the representative graph model is obtained, the importance vector is obtained by encoding the representative graph model according to the pre-learned pattern estimation method (S62). Then, the importance vector is decoded according to the pre-learned pattern recovery method to obtain a weighted relationship model graph having a weight (w _{i ) according to the importance of each node vector (n i} ^{t+1) of the relationship graph model as a node (S63)} ). When the weight relationship model graph is obtained, a correction relationship model graph is obtained by weighting the _{weight (w i} ) of the corresponding node of the weight relationship model graph to each _{of a plurality of node vectors (n i} ^{t+1 ) of the relationship graph model (} S64).

Referring back to FIG. 11 , it is determined whether the face recognition apparatus needs to be trained ( S70 ). If it is determined that it has already been learned, the correction relation model graph is mapped with identifiers for a plurality of subjects in the facial feature database 700 and similarity is compared with a plurality of reference relation graph models that are pre-stored correction relation model graphs (S81). Then, it is determined whether a similar reference relationship graph model having a similarity greater than or equal to a predetermined reference similarity is searched (S82). If the similar reference relationship graph model is searched for, an identifier mapped to the searched reference relationship graph model is output to perform face recognition (S83). However, if the similar reference relationship graph model is not searched, it is determined that the corresponding face image has not been searched and a face recognition failure is output (S84).

On the other hand, if it is determined that the face recognition device needs to be learned, learning is performed (S90). In the learning step (S90), first, the anchor, which is the face image of the subject, which is the reference of the face image, the positive which is the face image for the same subject as the anchor, and the negative which is the face image for the subject different from the anchor, respectively, correction relation The model graph is acquired and stored (S91). And on the acquired common space anchor and the positive node vector distance between _{^{(x i a, x i p}} ) the distance between the (s _i ^p) and the anchor and a negative node vector _{^{(x i a, x i n}} ) (s i n ) are calculated based on the degree of similarity (S92). The distance between the calculated anchor and the positive node vector (x _i ^a , x _i ^p _{) (s i} ^p ) and the distance between the anchor and the negative node vector (x _i ^a , x _i ⁿ ) (s _i ⁿ ) A loss is calculated based on the ratio and backpropagated to learn about the face recognition device (S93). _{In this case, a loss (L tripletconditional} ) may be calculated as in Equation (5) by additionally applying a margin (m) to perform robust face recognition by slowing the face recognition sensitivity of the face recognition apparatus.

The method according to the present invention may be implemented as a computer program stored in a medium for execution by a computer. Here, the computer-readable medium may be any available medium that can be accessed by a computer, and may include all computer storage media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data, and read dedicated memory), RAM (Random Access Memory), CD (Compact Disk)-ROM, DVD (Digital Video Disk)-ROM, magnetic tape, floppy disk, optical data storage, and the like.

Although the present invention has been described with reference to the embodiment shown in the drawings, which is merely exemplary, those skilled in the art will understand that various modifications and equivalent other embodiments are possible therefrom.

Accordingly, the true technical protection scope of the present invention should be defined by the technical spirit of the appended claims.

100: heterogeneous image acquisition unit 200: element feature extraction unit
300: vector position and distance calculation unit 400: vector relationship modeling unit
500: Relational model update unit 510: Representative value calculation unit
520: encoder 530: decoder
540: update unit 600: face recognition unit
700: facial feature database 800: learning unit

Claims (18)

Element that divides the acquired face image regardless of the domain indicating the acquisition method into a plurality of regions according to a pre-learned pattern estimation method, and obtains a feature map composed of a plurality of feature vectors by estimating the pattern of each divided region feature extraction unit;
a vector position and distance calculator for calculating a position and a relative distance of each of the plurality of feature vectors in the feature map and combining them with a corresponding feature vector;
a vector relationship modeling unit that considers each of a plurality of feature vectors in which a position and a relative distance are combined as a plurality of node vectors on a virtual common space, and obtains a relationship graph model in which the relationship between the plurality of node vectors is expressed as an edge;
A relation in which a weight according to the semantic importance of each of the plurality of node vectors of the relationship graph model is obtained according to a pre-learned pattern estimation method, and a correction relationship graph model is obtained by weighting the obtained weights to each of the plurality of node vectors model update unit; and
and a face recognition unit for recognizing the subject of the obtained face image by comparing the similarity between the correction relation graph model and a plurality of reference relation graph models that are pre-stored correction relation model graphs to which the subject is mapped. According to claim 1, wherein the vector relationship modeling unit
Each of a plurality of feature vectors combined with a position and a relative distance is projected on a virtual common space as a plurality of node vectors, and a plurality of edges representing relationships between the plurality of node vectors according to the plurality of node vectors are studied. A heterogeneous face recognition device for generating the relation graph model by setting it based on the relation weight obtained by the The method of claim 2, wherein the vector relationship modeling unit
According to the number of set edges, the relationship propagation of updating the multiple node vectors to the next state by propagating the relationship to the node vector connected to each edge, and updating the edge again according to the state of the updated node vector is repeated a predetermined number of times. a heterogeneous face recognition device. The method of claim 3, wherein the vector relationship modeling unit
_{The edge (e i,j} ^t ) of the current state connected in the direction from the i-th node vector (n _i ^t ) to the j-th node vector (n _i ^t ) among all D node vectors of the current state (t) is expressed by the equation

(Where σ is the activation function, and W _e ^T is the transpose matrix of relation weights.)
obtained according to
the i-th node, then the state vector of the node vector (n _i ^{t + 1)} from a plurality of the edges of the current status (e _k, ^t _i) connected to the (n _i ^t) Equation

A heterogeneous face recognition device acquired according to The method of claim 2, wherein the relationship model update unit
a representative value calculator for obtaining a representative value of each of the plurality of node vectors in a predetermined manner to obtain a representative graph model expressed as a representative value;
an encoder that encodes the representative graph model according to a pre-learned pattern estimation method to obtain an importance vector indicating the importance of each of a plurality of node vectors of the relation graph model;
a decoder that receives the importance vector and decodes it according to a pre-learned pattern recovery method to obtain a weight relation model graph having a weight according to the importance of each node vector of the relation graph model as a node; and
and an updater configured to obtain the correction relation model graph by weighting a weight of a corresponding node of the weight relation model graph to each of the plurality of node vectors of the relation graph model. The method of claim 5, wherein the heterogeneous face recognition device comprises:
Further comprising a learning unit for learning the vector relation modeling unit and the relation model update unit,
the learning unit
Correction relationship model graph for each of the anchor, which is the face image of the subject, which is the standard of the face image applied as training data, the positive, which is the face image of another domain for the same subject as the anchor, and the negative, which is the face image of the subject, which is different from the anchor. obtains, and calculates the distance between the anchor and each of the positive node vectors in the common space and the distance between the anchor and the negative node vector respectively, based on the similarity, the calculated anchor and the positive A heterogeneous face recognition device that calculates and backpropagates a loss based on a ratio between the distance between each corresponding multiple node vector and the distance between the anchor and each negative node vector. The method of claim 6, wherein the learning unit
A margin (m) specified in the ratio between the distance (s _i ^p ) between the anchor and each corresponding number of node vectors of the positive and the distance ( s _i ^{n ) between the anchor and the corresponding plurality of node vectors of each negative} In addition, by applying the formula

(Where [] ₊ is a positive function that takes only positive values.)
A heterogeneous face recognition device that computes the loss (L _{tripletconditional ) according to} According to claim 1, wherein the heterogeneous face recognition device
Further comprising a facial feature database in which the plurality of reference relationship graph models are mapped to the identifier of the subject and stored,
The face recognition unit
When a reference relationship graph model having a similarity greater than or equal to a predetermined reference similarity is found with the correction relationship graph model, an identifier mapped to the searched reference relationship graph model is output as a face recognition result. According to claim 1, wherein the heterogeneous face recognition device
A heterogeneous face recognition device that recognizes a subject by receiving one of a color image, an infrared image, a thermal infrared image, or a sketch image according to a domain. In the heterogeneous face recognition method of a heterogeneous face recognition device,
Classifying the acquired face image irrespective of the domain indicating the acquisition method into a plurality of regions according to a pre-learned pattern estimation method, and estimating the pattern of each divided region to obtain a feature map composed of a plurality of feature vectors ;
calculating a position and a relative distance of each of the plurality of feature vectors in the feature map and combining them with a corresponding feature vector;
obtaining a relationship graph model in which each of a plurality of feature vectors combined with a position and a relative distance is regarded as a plurality of node vectors on a virtual common space, and relationships between the plurality of node vectors are expressed as edges;
Obtaining a weight according to the semantic importance of each of the plurality of node vectors of a relationship graph model according to a pre-learned pattern estimation method, and obtaining a corrected relationship graph model by weighting the obtained weights on each of the plurality of node vectors ; and
and recognizing the subject of the obtained face image by comparing the correction relation graph model with a plurality of reference relation graph models that are pre-stored correction relation model graphs to which the subject is mapped. The method of claim 10, wherein the obtaining of the relation graph model comprises:
Projecting each of a plurality of feature vectors combined with a position and a relative distance as a plurality of node vectors on a virtual common space; and
and generating the relationship graph model by setting a plurality of edges representing relationships between the plurality of node vectors according to the plurality of node vectors based on a relationship weight obtained by learning. 12. The method of claim 11, wherein obtaining the relationship graph model comprises:
Repeating a predetermined number of times and propagating the relationship to the node vector connected to each edge according to the set multiple edges, updating the multiple node vectors to the next state, and propagating the relationship back to the edge according to the state of the updated node vector. A heterogeneous face recognition method further comprising. 13. The method of claim 12, wherein propagating the relationship comprises:
_{The edge (e i,j} ^t ) of the current state connected in the direction from the i-th node vector (n _i ^t ) to the j-th node vector (n _i ^t ) among all D node vectors of the current state (t) is expressed by the equation

(Where σ is the activation function, and W _e ^T is the transpose matrix of relation weights.)
obtaining according to; and
the i-th node, then the state vector of the node vector (n _i ^{t + 1)} from a plurality of the edges of the current status (e _k, ^t _i) connected to the (n _i ^t) Equation

A heterogeneous face recognition method comprising acquiring according to The method of claim 11, wherein the obtaining of the correction relationship graph model comprises:
obtaining a representative graph model expressed as a representative value by obtaining a representative value of each of the plurality of node vectors in a predetermined manner;
encoding the representative graph model according to a pre-learned pattern estimation method to obtain an importance vector indicating the importance of each of a plurality of node vectors of the relation graph model;
receiving the importance vector and decoding it according to a pre-learned pattern recovery method to obtain a weight relationship model graph having a weight according to the importance of each node vector of the relationship graph model as a node; and
and obtaining the correction relation model graph by weighting the weight of the corresponding node of the weight relation model graph to each of the plurality of node vectors of the relation graph model. 15. The method of claim 14, wherein the heterogeneous face recognition method
further comprising a learning step,
The learning step is
Correction relationship model graph for each of the anchor, which is the face image of the subject, which is the standard of the face image applied as training data, the positive, which is the face image of another domain for the same subject as the anchor, and the negative, which is the face image of the subject, which is different from the anchor. obtaining a;
calculating a distance between an anchor and a corresponding plurality of node vectors of each positive and a distance between an anchor and a corresponding plurality of node vectors of each negative on the common space based on the similarity; and
A heterogeneous face comprising the steps of calculating and backpropagating a loss based on the ratio between the calculated distance between the anchor and each corresponding multiple node vector of the positive and the distance between the anchor and the corresponding multiple node vector of each negative Recognition method. 16. The method of claim 15, wherein the step of backpropagating
A margin (m) specified in the ratio between the distance (s _i ^p ) between the anchor and each corresponding number of node vectors of the positive and the distance ( s _i ^{n ) between the anchor and the corresponding plurality of node vectors of each negative} In addition, by applying the formula

(Where [] ₊ is a positive function that takes only positive values.)
A heterogeneous face recognition method that computes the loss (L _{tripletconditional ) according to} The method of claim 10, wherein recognizing the subject comprises:
searching for a reference relationship graph model having a similarity greater than or equal to a predetermined reference similarity to the correction relationship graph model among the plurality of reference relationship graph models that are mapped to the identifier of the subject of the face image and stored;
When a reference relationship graph model having a similarity greater than or equal to the reference similarity is found, outputting an identifier mapped to the searched reference relationship graph model as a face recognition result. 11. The method of claim 10, wherein the heterogeneous face recognition method
A heterogeneous face recognition method that recognizes a subject by receiving one of a color image, an infrared image, a thermal infrared image, or a sketch image according to a domain.

Images