KR102027786B1 - Method and system for recognizing face of user based on multiple images - Google Patents

Abstract

Disclosed are a face recognition and authentication method simultaneously using a plurality of face images and a system thereof. A face recognition method comprises the steps of: storing a template corresponding to various face images of a user in a database; performing a quality test on a face image photographing the user for authentication; preprocessing the face image which has passed the quality test; extracting feature points from the face image on which the preprocessing has been performed; and authenticating a face of the user on the basis of a template based on the extracted feature points and the template stored in the database.

Description

Face recognition and authentication method and system using multiple face images at the same time {METHOD AND SYSTEM FOR RECOGNIZING FACE OF USER BASED ON MULTIPLE IMAGES}

Embodiments of the present invention relate to a machine learning technology for recognizing and authenticating a user using a face image of a user obtained through a photographing apparatus such as a camera.

Techniques for extracting feature points from images acquired using a camera are used for personal authentication through face recognition. Face recognition and authentication systems are known to show a great difference in recognition and authentication performance depending on the light source environment and the algorithm used. In particular, when the camera is not good performance, such as a smartphone (smartphone) is exposed to a variety of light source environment it is known that the face recognition and authentication performance is greatly reduced. As such, face recognition and authentication techniques using existing machine learning methodologies do not perform well in environments where face recognition and authentication performance is degraded. Therefore, the latest machine learning methodologies such as deep learning can be used to improve performance. However, due to the low computational performance of smartphones, there are still aspects that are difficult to use everyday.

An element of the feature point vector is determined by the influence of various position values of the face image. The face image given to the system is not only moved back and forth, but also to the left and right. Its hard to suppress its influence. When learning is performed using various face images as inputs, the learning problem still exists according to the learning environment.

Therefore, even in an electronic device such as a smart phone, which is difficult to apply high computing performance such as deep learning, a technology for improving the performance of recognizing and authenticating a user's face is required.

Korean Patent Laid-Open Publication No. 10-2010-0073749 relates to an SIFT-based feature point extraction apparatus and method, and a face recognition system using the same. The density threshold value of a candidate pixel used for feature point extraction in response to an input image ( A technique of calculating a contrast threshold, generating a keypoint descriptor using candidate pixels above a density threshold among all candidate pixels, and increasing the number of extracted feature points is disclosed.

The present invention relates to a technology for recognizing and authenticating a face of a user by simultaneously using various face images corresponding to the same user in an environment having low computational performance such as a smartphone.

A face recognition method comprising: storing a template corresponding to various face images of a user in a database, performing a quality test on a face image captured by a user for authentication, and passing the quality test Preprocessing a face image, extracting a feature point from the face image on which the preprocessing has been performed, and using a template based on the extracted feature point and a template stored in the database. Authenticating.

According to one aspect, the step of performing the quality test may include determining a face image of a predetermined predetermined quality index or more for a plurality of different face images of the face of the user.

According to another aspect, the determining of the face image above the predetermined quality index, the dispersion, the average of brightness, the standard deviation of brightness calculated using a Laplacian of Gaussian (LoG) filter for the plurality of different face images And a face image equal to or greater than the predetermined quality index may be determined based on at least one of similarity between, and face images.

According to another aspect, the performing of the quality test may include rejecting the authentication of the user when the face image photographed for the user who wants the authentication does not pass the quality test.

According to another aspect, the authenticating of the face of the user may include converting feature points for each image extracted from a plurality of face images photographed for the user into an integrated feature point, and the integrated feature point and the stored feature. The method may include authenticating the face of the user based on a feature point corresponding to a template.

According to another aspect, the face recognition system may include a storage control unit for storing templates corresponding to various face images of a user in a database, and a quality test for performing a quality test on a face image photographed against a user for authentication. A test unit, a preprocessing unit for preprocessing the face image that has passed the quality test, a feature point extracting unit for extracting feature points from the face image on which the preprocessing has been performed, and a template based on the extracted feature points and the template It may include an authentication unit for authenticating the face of the user based on the template stored in the database.

Face recognition system The authentication unit converts a feature point for each image extracted from a plurality of face images photographed for the user into an integrated feature point, and based on the integrated feature point and the feature point corresponding to the stored template, You can authenticate your face.

The present invention, in a low computing performance environment such as a smart phone, by using a variety of face images corresponding to the same user at the same time to recognize and authenticate the user's face, thereby improving face recognition performance without high computational processing such as deep learning Can be. That is, the machine learning is designed so that various face images of the same user have a similar point vector, and the element distribution of the feature point vector when the misjudgment (ie, recognition error) for the same user occurs is constant and constant. By performing face recognition in consideration of the difference above level, it is possible to recognize user's face more accurately captured by cameras such as smartphones in various environments, and to authenticate users during security authentication such as mobile banking and payment. Can be.

1 is a conceptual diagram illustrating a method of recognizing a face of a user based on a plurality of face images according to an embodiment of the present invention.
2 is a block diagram illustrating an internal configuration of a face recognition system according to an exemplary embodiment of the present invention.
3 is a flowchart illustrating a face recognition method according to an embodiment of the present invention.
4 is a diagram illustrating an operation of converting feature points extracted from a plurality of face images into integrated feature points according to an embodiment of the present invention.
FIG. 5 is a diagram illustrating a graph for calculating element values of integrated feature points according to one embodiment of the present invention.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

The present embodiments relate to a technology for recognizing and authenticating a user's face using various face images of a specific user. In particular, the present invention extracts an integrated feature point, which is a single feature point vector, from a plurality of face images, and extracts the extracted feature points. And a technology for recognizing and authenticating a face of a user based on a template of a specific user stored in advance.

In the present embodiments, a "template" is a collection of feature points extracted from a face image of a user and may be used for face recognition and authentication of the user. For example, a user's face recognition and authentication may be performed by comparing a template corresponding to a face image photographed through a camera with a template previously registered in a database.

In the present embodiments, the “integrated feature point” may indicate that the feature points corresponding to a specific location extracted from each of the plurality of face images are integrated and converted into one feature point for use in face recognition. For example, when there are 10 face images, the j th position corresponds to the s feature points extracted from the first image, the s feature points extracted from the second image,?, And the s feature points extracted from the 10th image. A feature point obtained by converting ten feature points (ie, j-th feature points corresponding to each of the first to tenth images) into one feature point for face recognition may correspond to an integrated feature point.

1 is a conceptual diagram illustrating a method of recognizing a face of a user based on a plurality of face images according to an embodiment of the present invention.

Referring to FIG. 1, a face of a user may be divided into a process 101 of registering a face image of the user and a recognition and authentication process 102 in order to recognize and authenticate the user's face.

The registration process 101 is the same in a variety of facial expressions, such as smiley faces, frowned faces, surprised faces, wind-reminded faces, various angles, and various environments (am, afternoon, rainy days, snowy days, hot days, etc.). A process of registering and storing various face images photographed for a user's face in association with identifier information of the user in the database 130 for user authentication may be described.

The registration process 101 performs a process 110 on a plurality of images to be registered in the database 130, extracts a feature 120 from the preprocessed images, 120. And storing a template based on the extracted feature point in the database 130 in association with identifier information of the corresponding user.

The recognition and authentication procedure 102 is a process of performing a quality test on a plurality of images photographing a user's face for recognition and authentication of the user's face (140). The process of performing the pre-processing (150), the process of extracting the feature points from the pre-processed image (160), the process of integrating the extracted feature points into one by location (170), and a template based on the integrated feature points ) And comparing the template stored in the database 130 to provide a result of recognizing a user's face and providing a result of performing authentication.

2 is a block diagram illustrating an internal configuration of a face recognition system according to an embodiment of the present invention, and FIG. 3 is a flowchart illustrating a face recognition method according to an embodiment of the present invention.

The face recognition system 200 according to the present embodiment is a computing system configured to process instructions of a computer program for face recognition, and includes a processor 210, a bus 220, a network interface 230, a memory 240, and Database 250 may be included. The memory 240 may include an operating system 241 and a service providing routine 242. The processor 210 may include a storage control unit 211, a quality test unit 212, a preprocessor 213, a feature point extractor 214, and an authentication unit 215. In other embodiments, facial recognition system 200 may include more components than the components of FIG. 2. However, it is not necessary to clearly show most of the prior art components. For example, face recognition system 200 may include other components, such as a display or a transceiver.

The memory 240 is a computer-readable recording medium and may include a permanent mass storage device such as random access memory (RAM), read only memory (ROM), and a disk drive. In addition, the memory 240 may store program codes for the operating system 241 and the service providing routine 242. These software components may be loaded from a computer readable recording medium separate from the memory 240 using a drive mechanism (not shown). Such a separate computer-readable recording medium may include a computer-readable recording medium (not shown) such as a floppy drive, a disk, a tape, a DVD / CD-ROM drive, a memory card, and the like. In other embodiments, the software components may be loaded into the memory 240 via the network interface 230 rather than the computer readable recording medium.

The bus 220 may enable communication and data transmission between components of the face recognition system 200. The bus 220 may be configured using a high-speed serial bus, a parallel bus, a storage area network and / or other suitable communication technology.

The network interface 230 may be a computer hardware component for connecting the face recognition system 200 to a computer network. The network interface 230 may connect the face recognition system 200 to a computer network through a wireless or wired connection.

The database 250 may store and maintain various facial image templates of a plurality of users in association with identifier information of the corresponding user. That is, a plurality of templates based on feature points extracted from a plurality of face images for a specific user may be stored and maintained in the database 250 in association with identifier information (eg, ID) of the corresponding user.

The database 250 may be included in the face recognition system 200 or may exist as an external database built on another system separate from the face recognition system 200. The face recognition system 200 may be implemented in a platform form to provide face recognition and authentication of a user to a user terminal connected to a server, or an application for providing a face recognition and authentication of a user on the user terminal. That is, it may be implemented in the form of a service app.

The processor 210 may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input / output operations of the face recognition system 200. The instructions may be provided to the processor 210 by the memory 240 or the network interface 230 and via the bus 220. The processor 210 may be configured to execute program codes for the storage controller 211, the quality test unit 212, the preprocessor 213, the feature point extractor 214, and the authenticator 215. Such program code may be stored in a recording device such as memory 240.

The storage control unit 211, the quality test unit 212, the preprocessor 213, the feature point extractor 214, and the authentication unit 215 may be configured to perform the steps 310 to 350 of FIG. 3. have.

In operation 310, the storage control unit 211 registers and stores a template corresponding to various face images of the user in a database in association with identifier information of the corresponding user. Here, the template may represent a shape (eg, a face shape, etc.) based on the extracted feature points for recognizing and authenticating the user's face. In addition, a template of a face image in various environments (eg, weather, illumination, time, etc.), a face image of various expressions, and a face image at various shooting angles is not associated with a single face image for each user. Can be registered and stored in the database.

In step 320, the quality test unit 212 may perform a quality test on the face image photographed for the user who wants to authenticate. In this case, the image to be used for the quality test may be a plurality.

The quality test unit 212 may perform a quality test that determines a face image corresponding to a predetermined predetermined quality index or more among a plurality of user's face images photographed through a camera such as a smartphone. That is, the quality test may correspond to a screening process of a face image having a quality more suitable for face recognition and authentication of the user among different face images photographing the face of the same user.

For example, assuming that a template of a specific user is T, T may be expressed as Equation 1 below.

[Equation 1]

According to Equation 1, the template T of a specific user may be expressed as a set of feature points Y [1], Y [2],? In this case, n face images (that is, n face images photographed through a camera) may be input for face recognition and authentication of a user. Here, each face image may correspond to an image photographing a facial expression in a specific pose. Then, the quality test unit 212 may determine m face images having excellent quality based on the n face images. For example, the quality test unit 212 may determine m face images having excellent quality corresponding to a certain quality index among the plurality of face images based on the quality test function QT () as shown in Equation 2 below.

[Equation 2]

In Equation 2, QT () may represent a quality test function, X [n] represents n face images corresponding to a specific user, and Z [m] may represent m face images corresponding to a predetermined quality index or more. Here, m = n may correspond.

At this time, the quality test unit 212 is a variance, average of brightness, standard deviation of brightness, and face images calculated using a Laglacian of Gaussian (LoG) filter on a plurality of different face images photographed by a camera. A face image of at least a certain quality index may be determined based on at least one of the similarities between the two. For example, the quality test unit 212 may determine a face image having a sharpness or more by applying a LoG filter to a plurality of face images and calculating dispersion. Then, a face image having a mean and standard deviation of brightness greater than or equal to a certain mean and a standard deviation may be determined. In addition, the similarity between the plurality of face images may be measured and a face image having a similarity or more may be determined. In addition, the quality test unit 212 may determine a face image having a predetermined quality or more by variously mixing at least two or more of dispersion, average of brightness, standard deviation, and similarity.

For example, a face image that is clear based on variance and whose brightness and average deviation are bright may be selected as an image to be used for face recognition and authentication. This quality test is for not using a face image smaller than a certain quality index for recognition and authentication, and the number m of face images selected may be flexible. That is, each time the user attempts to recognize the face, the number of face images passing the quality test may vary. For example, when face recognition and authentication are attempted a week ago, the number of images that passed the quality test is five, and the number of images that passed the quality test today may be as flexible as ten. In this case, the quality test unit 212 may reject the authentication of the user when the face image photographed for the user who wants to authenticate does not pass the quality test.

In operation 330, the preprocessor 213 may perform preprocessing on the face image that has passed the quality test.

For example, the preprocessor 213 may perform preprocessing for extracting feature points from m images having passed the quality test among the n face images. That is, the preprocessor 213 may perform preprocessing such as converting a color image into a grayscale image to extract feature points representing a specific shape such as eyes, nose, mouth, and eyebrows from the face image.

In operation 340, the feature point extractor 214 may extract feature points for each face image from a plurality of face images on which preprocessing has been performed.

In operation 350, the authenticator 215 may recognize the user's face based on the extracted template based on the extracted feature point and the template stored in the database, and perform authentication on the user.

In operation 351, the authenticator 215 may integrate feature points for each image extracted from the plurality of face images photographed by the user. That is, the authenticator 215 may convert feature points for each image extracted from the plurality of face images that have passed the quality test into integrated feature points.

In operation 352, the authenticator 215 may authenticate the face of the user based on the integrated feature points and the feature points corresponding to the template stored in the database. That is, the authenticator 215 may recognize and authenticate the face of the user based on the converted integrated feature point and the feature point corresponding to the template stored in the database. Here, the feature point transformation may be performed based on the integrated function MG () expressed as in Equation 3 below.

[Equation 3]

In Equation 3, MG () may represent an integration function, and F [m] may represent a feature point extracted from m preprocessed face images passing the quality test. Y2 is an integrated feature point and can be used for comparison with the template T stored in the database. That is, Y2 is {F [1], F [2],... , F [m]} may represent one integrated feature point converted for face recognition and authentication of the user through the integration function MG ().

Hereinafter, an operation of incorporating feature points will be described in more detail with reference to FIG. 4.

4 is a diagram illustrating an operation of converting feature points extracted from a plurality of face images into integrated feature points according to an embodiment of the present invention.

4, {F [1], F [2],... , F [m]} may represent a set of feature points extracted from each of the m face images that have passed the quality test and are preprocessed. That is, F [1] is a set including feature points extracted from face image 1, F [2] is a set including feature points extracted from face image 2, and F [m] is a feature point extracted from m th face image. It can represent a containing set. Then, the authentication unit 215 is the feature points {F [1], F [2], ... for each of the m face images. , F [m]} can be converted to a set of integrated feature points.

For example, each feature point F [i] given to the integration function MG () may be represented by a vector of size s, and each element may be represented by a number capable of comparing sizes, as shown in Equation 4 below.

[Equation 4]

에 대해서, j번째 위치의 원소들을 모아 놓은 새로운 벡터 Y[j] = {F[1][j], F[2][j], …, F[m][j]}가 얻어질 수 있다. 즉, Y[j]는 m개의 얼굴 이미지 별로 존재하는 복수의 특징점들 중 각 이미지 별 j번째 위치에 해당하는 특징점을 모아서 포함하는 벡터를 나타낼 수 있다. 그러면, 모든 위치

에 대하여 {Y[1], Y[2], …, Y[s]}가 얻어질 수 있다. 각 벡터 Y[j]를 정렬하면 {Y[j][1], Y[j][2], …, Y[j][m]}가 되고, 즉,

로 표현될 수 있다. 위의 수학식 4의 이미지 별로 추출된 특징점들 중 특정 위치 j에 해당하는 이미지 별 특징점을 모아 통합한 통합 특징점으로 변환하는 것은 아래의 수학식 5와 같이 표현될 수 있다.According to equation (4), all

For a new vector Y [j] = {F [1] [j], F [2] [j],... , F [m] [j]} can be obtained. That is, Y [j] may represent a vector including a feature point corresponding to the j th position of each image among a plurality of feature points existing for each of m face images. Then all positions

With respect to {Y [1], Y [2],. , Y [s]} can be obtained. Align each vector Y [j] with {Y [j] [1], Y [j] [2],... , Y [j] [m]}, that is,

It can be expressed as. Among the feature points extracted for each image of Equation 4 above, the feature points for each image corresponding to a specific position j may be collected and converted into an integrated feature point, which may be expressed as Equation 5 below.

[Equation 5]

Then, the authenticator 215 may perform face recognition and authentication of the user based on the transformed feature points for each location, that is, the s integrated feature points and the feature points of the template stored in the database, based on Equation (5). In order to compare s integrated feature points and the feature points of the template, an element value of an integrated feature point in which a feature point of a specific location is integrated is calculated based on m images. The element value of the integrated feature point will be described with reference to FIG. 5. do.

FIG. 5 is a diagram illustrating a graph for calculating element values of integrated feature points according to one embodiment of the present invention.

Y [j] [m / 2] (second quartile, 50% point, 501) from Y [j] [m / 4] (first quartile, 25% point, 502) and Y [j] [3m / 4] ( If the distance between the third quartile, 75% point, and 503 occurs by more than a predetermined reference multiple (e.g., a times), the authenticator 215 determines that Y2 [j] = (Y [j] [m / 4] + Y [j] [3m / 4]) / 2 can be calculated as the element value of the integrated feature point corresponding to the j th position. At this time, if the distance does not occur more than a predetermined reference multiple (for example, a times), the authentication unit 215 is Y2 [j] = Y [j] [m / 2] (second quartile, 50% point) Can be calculated as the element value of the integrated feature point.

이거나

이면 통합 특징점의 원소값 Y2[j]는 아래의 수학식 6에 기초하여 계산될 수 있다.That is, referring to 510 and 520 of FIG. 5, d1 = Y [j] [m / 2]-Y [j] [m / 4], d2 = Y [j] [3m / 4]-Y [j] When you say [m / 2]

Or

The element value Y2 [j] of the integrated feature point may be calculated based on Equation 6 below.

[Equation 6]

이거나

이 아닌 경우, 즉, d1이 기준 배수(a)와 d2의 곱 이하이거나, d2가 기준 배수(a)와 d1의 곱 이하인 경우, 인증부(215)는 통합 특징점의 원소값을 아래의 수학식 7에 기초하여 계산할 수 있다.At this time,

Or

If not, i.e., if d1 is less than or equal to the product of the reference multiple (a) and d2, or if d2 is less than or equal to the product of the reference multiple (a) and d1, the authenticator 215 determines the element value of the integrated feature point as It can calculate based on 7.

[Equation 7]

The authenticator 215 may perform face recognition and authentication of the user based on the element values of the integrated feature points corresponding to the specific positions calculated based on Equation 7 and the element values of the feature points constituting the template T stored in the database. have.

Methods according to an embodiment of the present invention may be implemented in program instruction form that can be executed by various computer systems and recorded in a computer readable medium.

The apparatus described above may be implemented as a hardware component, a software component, and / or a combination of hardware components and software components. For example, the devices and components described in the embodiments are, for example, processors, controllers, arithmetic logic units (ALUs), digital signal processors, microcomputers, field programmable gate arrays (FPGAs). Can be implemented using one or more general purpose or special purpose computers, such as a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing device may be described as one being used, but one of ordinary skill in the art will appreciate that the processing device includes a plurality of processing elements and / or a plurality of types of processing elements. It can be seen that it may include. For example, the processing device may include a plurality of processors or one processor and one controller. In addition, other processing configurations are possible, such as parallel processors.

The software may include a computer program, code, instructions, or a combination of one or more of the above, and configure the processing device to operate as desired, or process it independently or collectively. You can command the device. The software may be distributed over networked computer systems so that they may be stored or executed in a distributed manner. Software and data may be stored on one or more computer readable recording media.

The method according to the embodiment may be embodied in the form of program instructions that can be executed by various computer means and recorded in a computer readable medium. The computer readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the media may be those specially designed and constructed for the purposes of the embodiments, or they may be of the kind well-known and available to those having skill in the computer software arts. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tape, optical media such as CD-ROMs, DVDs, and magnetic disks, such as floppy disks. Magneto-optical media, and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine code generated by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like.

Although the embodiments have been described by the limited embodiments and the drawings as described above, various modifications and variations are possible to those skilled in the art from the above description. For example, the described techniques may be performed in a different order than the described method, and / or components of the described systems, structures, devices, circuits, etc. may be combined or combined in a different form than the described method, or other components. Or even if replaced or substituted by equivalents, an appropriate result can be achieved.

Therefore, other implementations, other embodiments, and equivalents to the claims are within the scope of the claims that follow.

Claims (7)

In the face recognition method,
Storing a template corresponding to various face images of a user in a database;
Performing a quality test on the plurality of face images to be photographed, and measuring the quality of each of the face images to be photographed;
Comparing at least two predetermined quality indexes with each other to determine at least two of the photographed face images;
Preprocessing the determined face images;
Extracting a plurality of feature points from different positions in each of the preprocessed face images with respect to the preprocessed face images;
Converting the feature points into an integrated feature point; And
Authenticating the face of the user based on the template based on the integrated feature point and the stored template
Including,
The integrated feature point includes a plurality of element values respectively corresponding to the positions, and each of the element values is calculated based on at least any two of the same positions among the feature points. delete The method of claim 1,
Measuring each of the quality of the face images to be photographed,
The photographed face images are photographed based on at least one of variance calculated using a Laplacian of Gaussian (LoG) filter, an average of brightness, a standard deviation of brightness, and similarity between the photographed face images. Face recognition method for measuring the quality of face images respectively. The method of claim 1,
If at least any two qualities of the photographed face images are less than the predetermined quality index, rejecting the authentication of the user
Face recognition method further comprising. delete A storage controller configured to store a template corresponding to various face images of the user in a database;
A quality test is performed on a plurality of face images to be photographed, each of which measures the quality of the face images to be photographed, and compares the qualities with a predetermined predetermined quality index, thereby selecting at least two of the face images. A quality test unit for determining a dog;
A preprocessing unit for preprocessing the determined face images;
A feature point extracting unit for extracting a plurality of feature points from different positions in each of the preprocessed face images with respect to the preprocessed face images; And
An authentication unit converting the feature points into an integrated feature point and authenticating the face of the user based on the template and the stored template based on the integrated feature point
Including,
The integrated feature point includes a plurality of element values respectively corresponding to the positions, and each of the element values is calculated based on at least two of the same positions among the feature points. The method of claim 6,
The quality test unit,
The photographed face images are photographed based on at least one of variance calculated using a Laplacian of Gaussian (LoG) filter, an average of brightness, a standard deviation of brightness, and similarity between the photographed face images. Face recognition system for measuring the qualities of face images respectively.

Images