WO2024147540A1 - Method and system for registering image embeddings for face recognition - Google Patents

Abstract

Disclosed are a method and system for registering image embeddings for face recognition. The embedding registration method according to an embodiment may comprise the steps of: extracting embeddings respectively from a plurality of face images of a person to be registered for face recognition and generating a set of embeddings; constructing a representative embedding on the basis of the embeddings included in the set of embeddings; and registering the constructed representative embedding in a database in association with an identifier of the person.

Description

Method and system for registering image embedding for face recognition

The description below relates to a method and system for registering image embeddings for face recognition.

In a general face recognition method, embeddings of face images of the person to be recognized can first be extracted and registered in a database along with the person's identifier. Afterwards, the first embedding for the input face image is compared with the second embeddings registered in the database, and an identifier with the second embedding with the highest similarity is output, thereby providing an identifier for the person recognized for the input face image. can be provided. In this case, only the second embedding whose similarity to the first embedding is greater than a certain threshold can be recognized, and if there is no second embedding whose similarity to the first embedding is greater than a certain threshold, it corresponds to the input face image. The person doing this may be recognized as someone who is not registered in the database.

We provide an embedding registration method and system that can configure a representative embedding for the input data group of a person to be registered in face recognition and register it in a database.

An embedding registration method of a computer device including at least one processor, comprising extracting an embedding from each of a plurality of face images of a person to be registered for face recognition, using the at least one processor, to generate an embedding set. ; Constructing, by the at least one processor, a representative embedding based on embeddings included in the embedding set; and registering, by the at least one processor, the configured representative embedding in a database in association with the identifier of the person.

According to one side, the step of generating the embedding set may be characterized by extracting a feature vector as an embedding from each of the plurality of face images and generating a set of feature vectors as an embedding set.

According to another aspect, the step of configuring the representative embedding comprises configuring the center point in the feature space of the feature vectors, which is determined by calculating the average or weighted average in the feature space of the feature vectors, as the representative embedding. You can do this.

According to another aspect, the step of registering in the database may be characterized by registering the center point in the feature space in the database by linking it with the identifier of the person.

According to another aspect, configuring the representative embeddings includes learning a deep learning-based classifier through feature vectors corresponding to the embeddings; and obtaining the weights of the last fully connected layer of the learned deep learning-based classifier as a weighted average center point in the feature space for the feature vectors.

According to another aspect, the step of configuring the representative embedding comprises configuring the representative embedding through a weighted average of the embeddings included in the embedding set, wherein the representative embedding is extracted from a specific face image among the plurality of face images or the The weighted average may be calculated by assigning a weight to a specific feature among the features included in the embeddings.

According to another aspect, the embedding registration method provides the administrator with a function for selecting a specific face image among the plurality of face images or a specific feature among the features included in the embeddings by the at least one processor. Additional steps may be included. At this time, the step of configuring the representative embedding involves configuring the representative embedding through a weighted average of the embeddings included in the embedding set, using an embedding extracted from a specific face image selected through the function or an embedding selected through the function. The weighted average may be calculated by assigning weights to specific features.

According to another aspect, the embedding registration method includes generating statistics about attributes from the plurality of face images using a classifier that analyzes attributes about faces, by the at least one processor; providing, by the at least one processor, the generated statistics to an administrator; extracting, by the at least one processor, additional embeddings from additional face images input from the manager based on the statistics; and further registering the additional embedding in a database in association with the identifier of the person, by the at least one processor.

According to another aspect, the embedding registration method includes receiving, by the at least one processor, a face image of a person whose face is to be recognized; extracting, by the at least one processor, a first embedding of the input face image; And further comprising, by the at least one processor, searching the database for a representative embedding whose reliability for distinguishing from the first embedding is greater than or equal to a preset threshold value, and extracting an identifier of a person stored in connection with the searched representative embedding. You can.

According to another aspect, the embedding registration method includes selecting, by the at least one processor, an additional face image; simulating, by the at least one processor, a change in performance when registering an embedding extracted from the selected additional face image in the database; providing, by the at least one processor, the simulation results to an administrator; and further registering, by the at least one processor, in response to a decision from the administrator to register the additional face image, in association with the identifier of the person, an embedding extracted from the additional face image in a database. can do.

According to another aspect, in the step of selecting the additional face image, the identifier of a specific person is extracted from the database according to the similarity between the embeddings, so that the reliability of distinguishing the embedding among the face images with the recognized face is within a preset boundary value. It may be characterized by selecting the face image included in the range determined by the additional face image.

According to another aspect, the embedding registration method includes recognizing, by the at least one processor, a test face image for the person using the database; When the test face image is not recognized, by the at least one processor, analyzing properties of the test face image and properties of each of the plurality of face images using a classifier that analyzes properties of a face; determining, by the at least one processor, attributes of an additional facial image for recognition of the test facial image based on the analyzed attributes; and providing, by the at least one processor, information about attributes of the determined additional face image to an administrator.

According to another aspect, the embedding registration method includes recognizing, by the at least one processor, a facial image of the person using the database; and, by the at least one processor, when the classification reliability of the embedding extracted from the face image is within a range determined by a preset boundary value, linking with the identifier of the person using the embedding extracted from the face image. A step of updating the representative embedding stored in the database may be further included.

According to another aspect, the embedding registration method is such that when representative embeddings of people registered in the database are updated by the at least one processor, the difference between the existing representative embedding and the updated representative embedding is greater than or equal to a preset threshold, Alternatively, when the representative embeddings of a preset number or more of the persons registered in the database are updated, a step of notifying the manager of a change in performance due to the update of the representative embeddings may be further included.

A computer program stored on a computer-readable recording medium is provided in conjunction with a computer device to execute the method on the computer device.

Provided is a computer-readable recording medium on which a program for executing the above method on a computer device is recorded.

At least one processor implemented to execute computer-readable instructions, wherein an embedding is extracted from each of a plurality of face images of a person to be registered for face recognition, and an embedding set is generated by the at least one processor. and configuring a representative embedding based on the embeddings included in the embedding set, and registering the constructed representative embedding in a database in association with the identifier of the person.

In face recognition, by constructing a representative embedding for the input data group of the person to be registered and registering it in the database, face recognition performance can be improved while reducing the amount of embedding data registered in the database.

1 is a diagram illustrating an example of a network environment according to an embodiment of the present invention.

Figure 2 is a block diagram showing an example of a computer device according to an embodiment of the present invention.

Figure 3 is a flowchart showing an example of an embedding registration method in one embodiment of the present invention.

Figure 4 is a diagram showing an example of a person and person images according to an embodiment of the present invention.

Figure 5 is a diagram illustrating an example of a process for extracting embeddings from a person image, according to an embodiment of the present invention.

Figure 6 is a diagram illustrating an example of configuring representative embedding in an embodiment of the present invention.

Figure 7 is a diagram showing an example of an average in a feature space according to an embodiment of the present invention.

Figure 8 is a flowchart showing an example of a process for recommending additional data, according to an embodiment of the present invention.

Figure 9 is a flowchart showing an example of a process for selecting and registering an additional face image, according to an embodiment of the present invention.

Figure 10 is a diagram illustrating an example of a process for determining attributes of an additional face image for a person, according to an embodiment of the present invention.

Figure 11 is a flowchart illustrating an example of a process for updating embeddings registered in a database, according to an embodiment of the present invention.

Figure 12 is a flowchart showing an example of a face recognition process in one embodiment of the present invention.

Figures 13 and 14 are graphs for comparing the performance of a face recognition method according to the prior art and the performance of a face recognition method according to an embodiment of the present invention.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings.

The embedding registration system according to embodiments of the present invention may be implemented by at least one computer device. At this time, the computer program according to an embodiment of the present invention may be installed and driven in the computer device, and the computer device may perform the embedding registration method according to the embodiments of the present invention under the control of the driven computer program. . The above-described computer program can be combined with a computer device and stored in a computer-readable recording medium to execute the embedding registration method on the computer.

1 is a diagram illustrating an example of a network environment according to an embodiment of the present invention. The network environment in FIG. 1 shows an example including a plurality of electronic devices 110, 120, 130, and 140, a plurality of servers 150 and 160, and a network 170. Figure 1 is an example for explaining the invention, and the number of electronic devices or servers is not limited as in Figure 1. In addition, the network environment in FIG. 1 only explains one example of environments applicable to the present embodiments, and the environment applicable to the present embodiments is not limited to the network environment in FIG. 1.

The plurality of electronic devices 110, 120, 130, and 140 may be fixed terminals or mobile terminals implemented as computer devices. Examples of the plurality of electronic devices 110, 120, 130, and 140 include smart phones, mobile phones, navigation devices, computers, laptops, digital broadcasting terminals, PDAs (Personal Digital Assistants), and PMPs (Portable Multimedia Players). ), tablet PC, etc. For example, in FIG. 1, the shape of a smartphone is shown as an example of the electronic device 110. However, in embodiments of the present invention, the electronic device 110 actually communicates with other devices through the network 170 using a wireless or wired communication method. It may refer to one of various physical computer devices capable of communicating with electronic devices 120, 130, 140 and/or servers 150, 160.

The communication method is not limited, and may include not only a communication method utilizing a communication network that the network 170 may include (for example, a mobile communication network, wired Internet, wireless Internet, and a broadcast network), but also short-range wireless communication between devices. For example, the network 170 may include a personal area network (PAN), a local area network (LAN), a campus area network (CAN), a metropolitan area network (MAN), a wide area network (WAN), and a broadband network (BBN). , may include one or more arbitrary networks such as the Internet. Additionally, the network 170 may include any one or more of network topologies including a bus network, star network, ring network, mesh network, star-bus network, tree or hierarchical network, etc. Not limited.

Each of the servers 150 and 160 is a computer device or a plurality of computers that communicate with a plurality of electronic devices 110, 120, 130, 140 and a network 170 to provide commands, codes, files, content, services, etc. It can be implemented with devices. For example, the server 150 may be a system that provides services to a plurality of electronic devices 110, 120, 130, and 140 connected through the network 170.

Figure 2 is a block diagram showing an example of a computer device according to an embodiment of the present invention. Each of the plurality of electronic devices 110, 120, 130, and 140 described above or each of the servers 150 and 160 may be implemented by the computer device 200 shown in FIG. 2.

As shown in FIG. 2, this computer device 200 may include a memory 210, a processor 220, a communication interface 230, and an input/output interface 240. The memory 210 is a computer-readable recording medium and may include a non-permanent mass storage device such as random access memory (RAM), read only memory (ROM), and a disk drive. Here, non-perishable large-capacity recording devices such as ROM and disk drives may be included in the computer device 200 as a separate permanent storage device that is distinct from the memory 210. Additionally, an operating system and at least one program code may be stored in the memory 210. These software components may be loaded into the memory 210 from a computer-readable recording medium separate from the memory 210. Such separate computer-readable recording media may include computer-readable recording media such as floppy drives, disks, tapes, DVD/CD-ROM drives, and memory cards. In another embodiment, software components may be loaded into the memory 210 through the communication interface 230 rather than a computer-readable recording medium. For example, software components may be loaded into memory 210 of computer device 200 based on computer programs installed by files received over network 170.

The processor 220 may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input/output operations. Commands may be provided to the processor 220 by the memory 210 or the communication interface 230. For example, processor 220 may be configured to execute received instructions according to program code stored in a recording device such as memory 210.

The communication interface 230 may provide a function for the computer device 200 to communicate with other devices (eg, the storage devices described above) through the network 170. For example, a request, command, data, file, etc. generated by the processor 220 of the computer device 200 according to a program code stored in a recording device such as memory 210 is transmitted to the network ( 170) and can be transmitted to other devices. Conversely, signals, commands, data, files, etc. from other devices may be received by the computer device 200 through the communication interface 230 of the computer device 200 via the network 170. Signals, commands, data, etc. received through the communication interface 230 may be transmitted to the processor 220 or memory 210, and files, etc. may be stored in a storage medium (as described above) that the computer device 200 may further include. It can be stored as a permanent storage device).

The input/output interface 240 may be a means for interfacing with the input/output device 250. For example, input devices may include devices such as a microphone, keyboard, or mouse, and output devices may include devices such as displays and speakers. As another example, the input/output interface 240 may be a means for interfacing with a device that integrates input and output functions, such as a touch screen. At least one of the input/output devices 250 may be configured as one device with the computer device 200. For example, like a smart phone, a touch screen, microphone, speaker, etc. may be included in the computer device 200.

Additionally, in other embodiments, computer device 200 may include fewer or more components than those of FIG. 2 . However, there is no need to clearly show most prior art components. For example, the computer device 200 may be implemented to include at least some of the input/output devices 250 described above, or may further include other components such as a transceiver, a database, etc.

Figure 3 is a flowchart showing an example of an embedding registration method in one embodiment of the present invention. The embedding registration method according to this embodiment can be performed by the computer device 200 previously described with reference to FIG. 2. At this time, the processor 220 of the computer device 200 may be implemented to execute control instructions according to the code of an operating system included in the memory 210 or the code of at least one computer program. Here, the processor 220 causes the computer device 200 to perform steps 310 to 330 included in the method of FIG. 3 according to control instructions provided by code stored in the computer device 200. can be controlled.

In step 310, the computer device 200 may generate an embedding set by extracting an embedding from each of a plurality of face images of a person to be registered for face recognition. For example, the computer device 200 may extract a feature vector as an embedding from each of a plurality of face images and generate a set of feature vectors as an embedding set. In other words, the embedding may be a feature vector extracted from a person's face image. Extraction of these feature vectors can be done using a previously learned machine learning-based facial feature extraction model. Since the technology itself for extracting features from images using machine learning-based learning models is already well known, detailed explanations will be omitted. When n face images are input for a person to be registered, n embeddings may be extracted to create an embedding set.

In step 320, the computer device 200 may construct a representative embedding based on the embeddings included in the embedding set. For example, the computer device 200 may configure the center point in the feature space of the feature vectors, which is determined by calculating the average or weighted average of the feature vectors, as a representative embedding. If a 512-dimensional feature vector is generated for each of n face images, data of size nХ512 is generated. On the other hand, when one central point in the feature space is configured as a representative embedding as in this embodiment, the required data can be reduced to data of 512 dimensions.

In one embodiment, the computer device 200 may obtain a weighted average center point through additional learning using a classifier for feature vectors. In this case, the computer device 200 can learn a deep learning-based classifier through feature vectors corresponding to embeddings included in the embedding set. In this case, the computer device 200 may obtain the weights of the last fully connected layer of the learned deep learning-based classifier as the weighted average center point in the feature space for the feature vectors. Depending on the embodiment, the results of L2 normalization of the weights may be used as a weighted average center point.

Meanwhile, as will be explained later, weights may be given in embedding units (face image units) or feature units. To this end, the computer device 200 constructs a representative embedding through a weighted average of the embeddings included in the embedding set, and uses an embedding extracted from a specific face image among a plurality of face images or a specific feature among the features included in the embeddings. You can calculate a weighted average by assigning weights. At this time, the administrator may be provided with a function to select embedding or features for assigning weight. In other words, the computer device 200 may provide the administrator with a function for selecting a specific feature among features included in a specific face image or embeddings among a plurality of face images. In this case, the computer device 200 configures the representative embedding through a weighted average of the embeddings included in the embedding set, and assigns weight to the embedding extracted from a specific face image selected through a function provided to the administrator or to the administrator. The provided function allows you to calculate a weighted average by assigning weight to specific selected features.

In step 330, the computer device 200 may register the constructed representative embedding in a database in association with the identifier of the person wishing to register. For example, the computer device 200 may link the central point in the feature space with the person's identifier and register it in a database. When representative embeddings such as these center points are used for face recognition, not only can the size of required data be reduced as described above, but face recognition performance can be improved through the reduced data. Additionally, depending on the embodiment, additional data in addition to the representative embedding may be used, and in this case, face recognition performance can be further improved. The improved face recognition performance will be explained in more detail later.

Figure 4 is a diagram showing an example of a person and person images according to an embodiment of the present invention. Figure 4 shows an example in which a plurality of face images 412, 422, and 432 exist for each of a plurality of people (person A (411), person B (421), person C (431), and so on. . At this time, a, b, and c, which represent the number of face images for each of person A (411), person B (421), and person C (431), may be substantially natural numbers of 2 or more.

Figure 5 is a diagram illustrating an example of a process for extracting embeddings from a person image, according to an embodiment of the present invention. In FIG. 5, each of the plurality of face images 412 of person A 411 is input into the machine learning-based facial feature extraction model 510 to generate embeddings 520 for each of the plurality of face images 412. An example of extraction is shown. In other words, the computer device 200 may extract embeddings 520 for each of the plurality of face images 412 using the machine learning-based facial feature extraction model 510. Previously, each of the embeddings 520 was stored in conjunction with the identifier of person A 411 and used for face recognition. However, in embodiments of the present invention, representative embeddings of these embeddings 520 can be configured and utilized. .

Figure 6 is a diagram showing an example of configuring representative embedding in one embodiment of the present invention. Figure 6 shows an example of configuring a representative embedding 610 by calculating the average or weighted average of the embeddings 520 of person A (411). Each of the embeddings 520 may be composed of a vector of a plurality of features, and the computer device 200 may obtain the value of each dimension of the representative embedding 610 by calculating the average or weighted average of the features of the same dimension. . The computer device 200 may register the constructed representative embedding 610 in the database 630 by linking it with the identifier 620 of person A 411.

At this time, the weight for the weighted average may be given on an embedding unit or a feature unit. Assigning weight in units of embeddings may mean assigning weights to each of a plurality of features of the embedding extracted from the selected face image, and assigning weights in units of features may mean assigning weights to the same dimension included in the embeddings 520. This may mean assigning weight to each of the features. In other words, assigning weights in units of embedding means that the more similar it is to a specific face image among the face images 412 of person A 411, the higher the similarity (classification reliability) is given so that face recognition can be detected more precisely. You can. Additionally, assigning weights by feature may mean that when comparing faces, the more similar a specific part of the face is, the higher the similarity (classification reliability) will be given to proceed with face recognition.

Here, segmentation reliability may refer to the degree to which a deep learning model classifies a face image (for example, classification into a characteristic person), and may correspond to the similarity calculated between embeddings. Since the technology for calculating similarity between embeddings for face recognition is already well known, detailed explanation is omitted.

Figure 7 is a diagram showing an example of an average in a feature space according to an embodiment of the present invention. Figure 7 shows two-dimensional simplified embeddings displayed on a two-dimensional feature space 800 to aid understanding of the invention. The first dotted oval 810 represents two-dimensional embeddings of the first person, the second dotted oval 820 represents two-dimensional embeddings of the second person, and the third dotted oval 830 represents two-dimensional embeddings of the third person. It is showing. At this time, the 2D embeddings may correspond to coordinates in the 2D feature space, and the average of these 2D embeddings may correspond to the centroid of the 2D embeddings in the 2D feature space. In other words, the coordinates of each of the central points 811, 821, and 831 in FIG. 7 may correspond to the 2D representative embedding of the first person, the 2D representative embedding of the second person, and the 2D representative embedding of the third person.

Figure 8 is a flowchart showing an example of a process for recommending additional data, according to an embodiment of the present invention. The additional data recommendation process according to this embodiment may be included in the embedding registration method described with reference to FIG. 4 and performed by the computer device 200. At this time, the processor 220 of the computer device 200 may be implemented to execute control instructions according to the code of an operating system included in the memory 210 or the code of at least one computer program. Here, the processor 220 causes the computer device 200 to perform steps 810 to 840 included in the process of FIG. 8 according to control instructions provided by code stored in the computer device 200. can be controlled.

In step 810, the computer device 200 may generate statistics on attributes from a plurality of face images using a classifier that analyzes attributes of the face. Attributes may include attribute values for facial expression, age, pose, whether a mask is worn, etc., which are analyzed through facial images.

In step 820, the computer device 200 may provide the generated statistics to the administrator. Through these statistics, administrators can determine which attribute face images are lacking and decide whether to utilize additional facial images with certain attributes.

In step 830, the computer device 200 may extract additional embeddings from additional face images input from the administrator based on statistics. Additional embeddings can be used as data to supplement face images with insufficient attributes.

In step 840, the computer device 200 may further register additional embeddings in the database by linking them to the person's identifier. In this case, each of the representative embeddings and additional embeddings may be stored in the database in association with the person's identifier.

Depending on the embodiment, the computer device 200 may provide the administrator with feedback on performance changes as additional embeddings are registered in the database.

Meanwhile, it was previously explained that weights can be assigned by embedding unit or feature unit. At this time, the embedding or feature to be weighted may be selected by the computer device 200, or may be manually selected by an administrator. To this end, a user interface for assigning weights may be provided to the administrator. In this case, the user interface may include a function that allows the administrator to select a face image to which weighting is to be assigned among a plurality of face images in order to select an embedding unit. Additionally, to select a feature unit, the computer device 200 may present the attributes described above through a user interface. In other words, the computer device 200 may provide a function for determining whether to assign weight to each attribute of a face through a user interface. For example, the computer device 200 may include “1) expression: smile, 2) age: 50s, 3) pose: [pan: 45eh, tilt: 0 degrees], 4) mask:” as attributes for a specific face image. Attributes and attribute values of 1) to 4), such as "radish", can be presented to the administrator through the user interface. In this case, the administrator can select the attribute for which the weight is to be reflected or the attribute for which the weight is not to be reflected through the user interface. Each attribute may be linked to at least one of the features of the embedding, and the features linked to the attribute may be determined according to the attribute selected by the administrator. In this case, weights may be assigned to the determined features.

Additionally, through this, it will be easy to understand that weighting can be applied to both embedding units and feature units simultaneously. For example, through the above-described user interface, the administrator can select a specific image among a plurality of face images, select a specific attribute among the attributes of the selected specific image, and assign weight to a specific feature of a specific embedding. . As a more specific example, a case where an administrator selects to assign weight to a first face image among a plurality of face images may be considered. At this time, if it is determined that bias will occur due to the first attribute of the first face image (for example, a pose looking to the left), no weight is given to the features linked to the first attribute. It becomes possible. Through this, bias due to specific attributes can be minimized.

Figure 9 is a flowchart showing an example of a process for selecting and registering an additional face image, according to an embodiment of the present invention. The additional facial image selection and registration process according to this embodiment may be included in the embedding registration method described with reference to FIG. 4 and performed by the computer device 200. At this time, the processor 220 of the computer device 200 may be implemented to execute control instructions according to the code of an operating system included in the memory 210 or the code of at least one computer program. Here, the processor 220 causes the computer device 200 to perform steps 910 to 940 included in the process of FIG. 9 according to control instructions provided by code stored in the computer device 200. can be controlled.

At step 910, computer device 200 may select additional facial images. Additional facial images may be manually selected and entered by a system administrator, or may be automatically selected by computer device 200, as in step 910. For example, the computer device 200 extracts the identifier of a specific person from the database according to the similarity between the embeddings, and the recognition reliability of the embedding among the face images is included in the range determined by the preset boundary value. Images can be selected for additional face images.

As described above, segmentation reliability can refer to the degree to which a deep learning model classifies a face image (for example, classifying it into a feature person) and can correspond to the similarity calculated between embeddings. At this time, the fact that the segmentation reliability is close to the boundary value for face classification may mean that the similarity between embeddings is not high. For example, if the similarity between the first embedding of the first facial image for the first person and the second embedding registered for the first person is 0.01 greater than the threshold (e.g., 0.5), the first facial image Although the corresponding person may be recognized as the first person, it can be seen that the reliability of this recognition is not high. In this case, when the first embedding of the first face image is stored in the database, the reliability of distinguishing face images similar to the first face image can be increased. Accordingly, the computer device 200 may select face images whose classification reliability falls within a range determined by a preset boundary value (for example, a range of 0.5 to 0.55) as candidates for additional face images.

In step 920, the computer device 200 may simulate performance changes when registering embeddings extracted from selected additional face images in a database. In other words, rather than registering all the embeddings of the additional face images selected in step 920, the computer device 200 may first simulate the performance change when registering the embeddings of the additional face images in the database.

In step 930, the computer device 200 may provide simulation results to the manager. Based on these simulation results, the administrator can decide whether it would be advantageous to register the embedding of the additional face image in the database even if it requires performance changes.

In step 940, in response to the administrator's decision to register an additional face image, the computer device 200 may further register the embedding extracted from the additional face image in the database in association with the person's identifier. In this case, each of the representative embedding and the embedding extracted from the additional face image may be stored in the database in association with the person's identifier.

Figure 10 is a diagram illustrating an example of a process for determining attributes of an additional face image for a person, according to an embodiment of the present invention. The process of determining additional facial image attributes according to this embodiment may be included in the embedding registration method described with reference to FIG. 4 and performed by the computer device 200. At this time, the processor 220 of the computer device 200 may be implemented to execute control instructions according to the code of an operating system included in the memory 210 or the code of at least one computer program. Here, the processor 220 causes the computer device 200 to perform steps 1010 to 1040 included in the process of FIG. 10 according to control instructions provided by code stored in the computer device 200. can be controlled.

In step 1010, the computer device 200 may recognize a test face image of a person using a database. For example, the computer device 200 searches for the first embedding of the test face image and an embedding (representative embedding and/or additional embedding) whose distinction reliability is greater than or equal to a preset boundary value and extracts the identifier of the person stored in connection with the searched embedding. , the person corresponding to the test face image can be recognized through the database.

In step 1020, if the test face image is not recognized, the computer device 200 may analyze the properties of the test face image and the properties of each of the plurality of face images using a classifier that analyzes the properties of the face. . In other words, if a specific test face image for a person already registered in the database is not recognized using that data, this may mean that the embeddings (representative embeddings and/or additional embeddings) registered in the database are incomplete. Additional facial images may be required.

In step 1030, the computer device 200 may determine properties of an additional face image for recognition of the test face image based on the analyzed properties. In other words, the computer device 200 may determine whether an additional face image of any attribute is needed for the person in the test face image to be recognized according to the analyzed attribute.

In step 1040, the computer device 200 may provide information about the properties of the determined additional face image to the manager. In this case, the administrator can input an additional face image of the provided attribute into the computer device 200, through which the computer device 200 further registers the embedding of the additional face image in the database as an additional embedding, thereby adding the embedding registered in the database. can complement.

Registering the embeddings of these additional face images may be used to determine face images with additional required attributes when registering a new person, or may be used to supplement the embeddings registered for an existing person. When registering a new person, a plurality of face images of the new person are classified into a test face image and a registered face image, a representative embedding is registered using the registered face image as in the embedding registration method according to the embodiment of FIG. 3, and then tested. For each face image, it is possible to determine which attribute of the face image is lacking (should be registered further) through an additional face image attribute determination process as shown in FIG. 10.

Figure 11 is a flowchart illustrating an example of a process for updating embeddings registered in a database, according to an embodiment of the present invention. The embedding update process according to this embodiment may be included in the embedding registration method described with reference to FIG. 4 and performed by the computer device 200. At this time, the processor 220 of the computer device 200 may be implemented to execute control instructions according to the code of an operating system included in the memory 210 or the code of at least one computer program. Here, the processor 220 causes the computer device 200 to perform steps 1110 to 1130 included in the process of FIG. 11 according to control instructions provided by code stored in the computer device 200. can be controlled.

In step 1110, the computer device 200 may recognize a face image of a person using a database. For example, the computer device 200 searches for the first embedding of the face image and an embedding (representative embedding and/or additional embedding) whose separation reliability is greater than or equal to a preset threshold value, and extracts the identifier of the person stored in connection with the searched embedding, The person corresponding to the face image can be recognized through the database.

In step 1120, if the classification reliability of the embedding extracted from the face image is within the range determined by the preset boundary value, the computer device 200 uses the embedding extracted from the face image to link it with the identifier of the person. Representative embeddings stored in the database can be updated. As described above, for example, if the similarity between the first embedding of the first face image for the first person and the second embedding registered for the first person is 0.01 greater than the boundary value (e.g., 0.5), the 1 It can be seen that the person corresponding to the face image can be recognized as the first person, but the classification reliability for this recognition is not high. This creates a request for updating the embedding currently registered in the database.

In this way, the computer device 200 includes the classification reliability in a range determined by a preset boundary value (for example, the range of boundary value + n, when the boundary value is 0.5 and n is 0.05, the range is 0.5 to 0.55). If so, the representative embedding stored in the database can be updated. For example, if the representative embedding is composed of the average value of 5 embeddings, the representative embedding can be updated with the average value of 6 embeddings that further include an embedding extracted from a new face image of the same person. Here, the new face image may correspond to the face image described in step 1110.

In step 1130, the computer device 200 is updated when the representative embedding of the people registered in the database is updated and the difference between the existing representative embedding and the updated representative embedding is greater than or equal to a preset threshold, or the representative embedding of the people registered in the database is updated. If the representative embeddings of more than a set number of people are updated, the manager can be notified of performance changes by updating the representative embeddings. In other words, the computer device 200 notifies the administrator of a performance change when the change in representative embeddings due to an update is greater than a threshold, or when the representative embeddings registered for more than a preset number of people are updated, the computer device 200 reports a performance change. You can notify the administrator.

Figure 12 is a flowchart showing an example of a face recognition process in one embodiment of the present invention. The face recognition process according to this embodiment may be performed by the computer device 200 after embedding registration. At this time, the computer device 200 performing the face recognition process of FIG. 12 may be the same device as the computer device 200 performing the embedding registration process of FIG. 4 or may be a different device. The processor 220 of the computer device 200 may be implemented to execute control instructions according to the code of an operating system included in the memory 210 or the code of at least one computer program. Here, the processor 220 is a computer device (200) such that the computer device 200 performs steps 1210 to 1230 included in the face recognition process of FIG. 12 according to a control command provided by a code stored in the computer device 200. 200) can be controlled.

In step 1210, the computer device 200 may receive an input face image of a person whose face is to be recognized. If the plurality of face images of the person described in step 310 of FIG. 4 were images for registering the person in the database, the face image input in step 1210 of FIG. 12 is the person corresponding to the face image in the database. This may be an image used to find out who this person is through the person's identifier.

In step 1220, the computer device 200 may extract the first embedding of the input face image. The process of extracting the first embedding from the face image may be the same as the process of extracting the previous embeddings.

In step 1230, the computer device 200 may search the database for a representative embedding whose reliability for distinguishing from the first embedding is greater than or equal to a preset threshold value and extract the identifier of the person stored in connection with the searched representative embedding. For example, the computer device 200 may compare the first embedding extracted from the input face image with each of the representative embeddings registered in the database and calculate the classification reliability for each of the representative embeddings. At this time, the computer device 200 can confirm a representative embedding whose classification reliability is greater than or equal to a preset boundary value, and can identify the person corresponding to the input face image by extracting the identifier of the person stored in connection with the confirmed representative embedding. There will be. If there is no representative embedding for which a classification reliability greater than a preset threshold value is obtained, the computer device 200 may classify the person corresponding to the input face image as a person not registered in the database.

Figures 13 and 14 are graphs for comparing the performance of a face recognition method according to the prior art and the performance of a face recognition method according to an embodiment of the present invention. Each graph in FIGS. 13 and 14 shows similarity in face recognition. At this time, in each graph, "Positive" represents a case where face recognition was attempted using a face image of a registered person, and "Negative" represents a case where face recognition was attempted using a face image of an unregistered person.

The graph in FIG. 13 shows a histogram when all embeddings of a plurality of face images of a person are registered in the database, and FIG. 14 shows a histogram when representative embeddings according to the average (weighted average) of the embeddings are registered in the database. It is showing. At this time, the average accuracy in the graph of FIG. 13 was 0.9174, and the average accuracy in the graph of FIG. 14 was 0.9258. In this way, as in the embodiments of the present invention, the average accuracy increased by about 0.008 in the case of Figure 14, where less data was registered by registering representative embeddings in the database. What is interesting is that when comparing the graph of FIG. 13 and the graph of FIG. 14, the number of cases with small similarity values increased in the case of “Positive”, but the similarity further decreased in the case of “Negative”. It is predicted that the average accuracy in the case of Figure 14 increases as the similarity decreases in the "Negative" case.

As such, according to embodiments of the present invention, representative embeddings for the input data group of a person to be registered in face recognition are configured and registered in the database, thereby improving face recognition performance while reducing the amount of embedding data registered in the database. You can do it.

The system or device described above may be implemented with hardware components or a combination of hardware components and software components. For example, devices and components described in embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), etc. , may be implemented using one or more general-purpose or special-purpose computers, such as a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. Additionally, a processing device may access, store, manipulate, process, and generate data in response to the execution of software. For ease of understanding, a single processing device may be described as being used; however, those skilled in the art will understand that a processing device includes multiple processing elements and/or multiple types of processing elements. It can be seen that it may include. For example, a processing device may include a plurality of processors or one processor and one controller. Additionally, other processing configurations, such as parallel processors, are possible.

Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing unit to operate as desired, or may be processed independently or collectively. You can command the device. Software and/or data may be used on any type of machine, component, physical device, virtual equipment, computer storage medium or device to be interpreted by or to provide instructions or data to a processing device. It can be embodied in . Software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., singly or in combination. The medium may continuously store a computer-executable program, or may temporarily store it for execution or download. In addition, the medium may be a variety of recording or storage means in the form of a single or several pieces of hardware combined. It is not limited to a medium directly connected to a computer system and may be distributed over a network. Examples of media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, And there may be something configured to store program instructions, including ROM, RAM, flash memory, etc. Additionally, examples of other media include recording or storage media managed by app stores that distribute applications, sites or servers that supply or distribute various other software, etc. Examples of program instructions include machine language code, such as that produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc.

As described above, although the embodiments have been described with limited examples and drawings, various modifications and variations can be made by those skilled in the art from the above description. For example, the described techniques are performed in a different order than the described method, and/or components of the described system, structure, device, circuit, etc. are combined or combined in a different form than the described method, or other components are used. Alternatively, appropriate results may be achieved even if substituted or substituted by an equivalent.

Therefore, other implementations, other embodiments and equivalents of the claims also fall within the scope of the following claims.

Claims (20)

1. In an embedding registration method of a computer device including at least one processor,

   generating an embedding set by extracting an embedding from each of a plurality of face images of a person to be registered for face recognition, by the at least one processor;

   Constructing, by the at least one processor, a representative embedding based on embeddings included in the embedding set; and

   Registering the configured representative embedding in a database in association with the identifier of the person, by the at least one processor

   Embedding registration method including.
2. According to paragraph 1,

   The step of generating the embedding set is,

   An embedding registration method characterized by extracting a feature vector as an embedding from each of the plurality of face images and generating a set of feature vectors as an embedding set.
3. According to paragraph 2,

   The step of configuring the representative embedding is,

   An embedding registration method, characterized in that configuring a center point in the feature space of the feature vectors, which is determined by calculating an average or weighted average in the feature space of the feature vectors, as the representative embedding.
4. According to paragraph 3,

   The step of registering in the database is,

   An embedding registration method characterized by registering the central point in the feature space in a database by linking it with the identifier of the person.
5. According to paragraph 1,

   The step of configuring the representative embedding is,

   Learning a deep learning-based classifier through feature vectors corresponding to the embeddings; and

   Obtaining the weights of the last fully connected layer of the learned deep learning-based classifier as a weighted average center point in the feature space for the feature vectors.

   An embedding registration method comprising:
6. According to paragraph 1,

   The step of configuring the representative embedding is,

   The representative embedding is constructed through a weighted average of the embeddings included in the embedding set, and a weight is given to an embedding extracted from a specific face image among the plurality of face images or a specific feature among the features included in the embeddings. An embedding registration method characterized by calculating a weighted average.
7. According to paragraph 1,

   Providing, by the at least one processor, a function for selecting a specific face image among the plurality of face images or a specific feature among the features included in the embeddings to the manager.

   It further includes,

   The step of configuring the representative embedding is,

   The representative embedding is constructed through a weighted average of the embeddings included in the embedding set, and the weighted average is calculated by weighting an embedding extracted from a specific face image selected through the function or a specific feature selected through the function. An embedding registration method characterized by calculating.
8. According to paragraph 1,

   generating statistics about attributes from the plurality of face images using a classifier that analyzes attributes about faces, by the at least one processor;

   providing, by the at least one processor, the generated statistics to an administrator;

   extracting, by the at least one processor, additional embeddings from additional face images input from the manager based on the statistics; and

   further registering, by the at least one processor, the additional embedding in a database in association with the identifier of the person

   An embedding registration method further comprising:
9. According to paragraph 1,

   receiving, by the at least one processor, a face image of a person whose face is to be recognized;

   extracting, by the at least one processor, a first embedding of the input face image; and

   Searching, by the at least one processor, from the database for a representative embedding whose reliability for distinguishing from the first embedding is greater than or equal to a preset threshold value, and extracting an identifier of a person stored in connection with the searched representative embedding.

   An embedding registration method further comprising:
10. According to paragraph 1,

    selecting, by the at least one processor, additional facial images;

    simulating, by the at least one processor, a change in performance when registering an embedding extracted from the selected additional face image in the database;

    providing, by the at least one processor, the simulation results to an administrator; and

    Further registering, by the at least one processor, in response to a decision from the administrator to register the additional face image, in association with the identifier of the person, an embedding extracted from the additional face image in a database.

    An embedding registration method further comprising:
11. According to clause 10,

    The step of selecting the additional face image is,

    As the identifier of a specific person is extracted from the database according to the similarity between embeddings, among the recognized face images, the face images whose classification reliability of the embedding is within the range determined by the preset boundary value are selected as the additional face images. An embedding registration method characterized by:
12. According to paragraph 1,

    Recognizing, by the at least one processor, a test face image of the person using the database;

    When the test face image is not recognized, by the at least one processor, analyzing properties of the test face image and properties of each of the plurality of face images using a classifier that analyzes properties of a face;

    determining, by the at least one processor, attributes of an additional facial image for recognition of the test facial image based on the analyzed attributes;

    providing, by the at least one processor, information about attributes of the determined additional facial image to an administrator.

    An embedding registration method further comprising:
13. According to paragraph 1,

    Recognizing, by the at least one processor, a facial image of the person using the database; and

    When the classification reliability of the embedding extracted from the face image is within a range determined by a preset boundary value by the at least one processor, the embedding extracted from the face image is used to link it with the identifier of the person. Updating the representative embedding stored in the database

    An embedding registration method further comprising:
14. According to paragraph 1,

    As the representative embeddings of the people registered in the database are updated by the at least one processor, when the difference between the existing representative embeddings and the updated representative embeddings is more than a preset threshold, or if the difference between the existing representative embeddings and the updated representative embeddings is more than a preset threshold, or When the representative embeddings of more than a number of people are updated, the step of notifying the manager of the performance change due to the update of the representative embeddings

    An embedding registration method further comprising:
15. A computer program stored in a computer-readable recording medium in combination with a computer device to cause the computer device to execute the method of any one of claims 1 to 14.
16. A computer-readable recording medium recording a computer program for executing the method of any one of claims 1 to 14 on a computer device.
17. At least one processor implemented to execute computer readable instructions

    Including,

    By the at least one processor,

    Generate an embedding set by extracting embeddings from each of a plurality of face images of a person to be registered for face recognition,

    Constructing a representative embedding based on the embeddings included in the embedding set,

    Registering the constructed representative embedding in a database by linking it with the identifier of the person.

    A computer device characterized by a.
18. According to clause 17,

    To generate the embedding set, by the at least one processor,

    Extracting a feature vector as an embedding from each of the plurality of face images and generating a set of feature vectors as an embedding set.

    A computer device characterized by a.
19. According to clause 17,

    To construct the representative embedding, by the at least one processor,

    Learn a deep learning-based classifier through feature vectors corresponding to the embeddings,

    Obtaining the weights of the last fully connected layer of the learned deep learning-based classifier as a weighted average center point in the feature space for the feature vectors.

    A computer device characterized by a.
20. According to clause 17,

    To construct the representative embedding, by the at least one processor,

    The representative embedding is constructed through a weighted average of the embeddings included in the embedding set, and a weight is given to an embedding extracted from a specific face image among the plurality of face images or a specific feature among the features included in the embeddings. Calculating a Weighted Average

    A computer device characterized by a.

Images