KR20160098581A - Method for certification using face recognition an speaker verification - Google Patents

Abstract

The present invention relates to a certificating method converging face recognition and speaker recognition, capable of performing certification by recognizing a face and a voice and converging the face and the voice. One embodiment of the present invention comprises: a face feature point extracting step of detecting a face image from the entire image and extracting a face feature vector from the detected face image; a voice feature point extracting step of detecting a voice from an audio and extracting voice feature data from the detected voice; a face similarity calculating step of calculating a face similarity between a registered feature vector which is previously registered and the face feature vector extracted from the face image; a voice similarity calculating step of calculating a voice similarity between a registered feature data which is previously registered and the voice feature data extracted from the voice; and an identity certification step of performing an identity certification by applying the face similarity and the voice similarity to an artificial neural network model.

Description

[0001] The present invention relates to a face recognition method and a speaker recognition method,

The present invention relates to an authentication method in which face recognition and speaker recognition are blended, and relates to a method for recognizing a face, recognizing a voice, and fusing it to perform authentication.

The 21st century is called the age of information. The reason is that because of the spread of the internet, collecting and analyzing the information that we want is simple and convenient, and our life is getting rich in various aspects including life. However, the use of the Internet has created a global network, and in the life that has become convenient, the important information of the individual is stolen or destroyed by others. As a result of this development of the Internet and IT technology, our lives have become comfortable and enriched, but personal information is leaked, shared, and damaged due to hacking and various cyber crimes.

In the past, a user password or PIN (Personal Identification Number) technique has been widely used for user authentication. However, this method has many risks and problems in authentication due to carelessness and has a problem that it can be used illegally through loss. Therefore, in the user authentication method using only the user password or PIN (Personal Identification Number), it is considered that the important information of the industry and the country can not be safely stored as well as the individual.

In order to solve these problems, there has been recently proposed a biometric method for verifying the identity of a person according to a physical or behavioral characteristic, which is a unique biometric information of an individual. (Biometrics), that is, biometrics technology using biometrics is emerging.

In particular, the development of a biometric recognition technology which does not have a sense of rejection by the user and which is convenient for the user is being actively promoted.

Korean Patent Publication No. 10-2006-0101244

An object of the present invention is to provide a face-feature vector and a face-feature vector, and to apply an authentication algorithm to an artificial neural network model that enables user authentication using these values. Another object of the present invention is to provide a stable authentication method without an authentication error.

An embodiment of the present invention relates to a facial feature point extraction process for detecting a facial image in a whole image and extracting a facial feature vector from the detected facial image; A voice feature point extracting step of detecting voice in audio and extracting voice feature data from the detected voice; A face similarity degree calculating step of calculating a face similarity degree between a registered feature vector registered in advance and a face feature vector extracted from the face image; A voice similarity degree calculating step of calculating a voice similarity degree between pre-registered registered characteristic data and voice specific data extracted from the voice; And an identity authentication process for applying the face similarity and voice similarity to the ANN model to perform identity authentication.

The method according to claim 1, wherein the facial feature point extracting step comprises: a face image detecting step of detecting a face image step by step in the entire image; A normalization process of normalizing the size and color of the detected face image to a set value; And a face feature vector extraction process for extracting a face feature vector from the normalized face image.

The stepwise detection of the face image can detect the face image step by step through the adaboost method.

The normalization process may include a size normalization process of converting the size of the detected face image into a predetermined size and outputting the size normalized face image.

After the size normalization process is completed, the tilted inclination angle and the inclination direction of the resized face image are grasped, and the face image is rotated in a direction opposite to the inclination direction by the inclination angle to output the resized normalized face image An angle normalization process.

In order to grasp the inclination angle in the angle normalization process, the size-converted face image is rotated at intervals of 2.5 degrees, valleys and edges in the vertical direction are extracted, and a vertical direction histogram in the valleys and edges in the vertical direction is calculated Process; And a step of determining a rotation angle when the vertical direction histogram calculated by rotating the resized face image at intervals of 2.5 degrees has the largest variance as an inclination angle, The brightness of the image pixel is lower than that of the surrounding pixels, the brightness difference between the image pixel and the surrounding pixels is larger than a predetermined threshold value, and the edge is the face contour of the resized face image .

And an illumination normalization process of removing the illumination effect from the normalized face image and outputting the illumination normalized face image after the angle normalization process.

Eliminating the illumination effect may remove the multiplicative moise and additive noise from the normalized face image and output it as an illuminated normalized face image.

(X, y) is the pixel function of the illuminated face image, I (x, y) is the pixel function of the illuminated face image, I (x, y) is the pixel function of the normalized face image from which the illumination effect is removed, E is the average of the pixel values of the normalized facial image, and VAR is the variance,

And outputs the calculated value.

The facial feature vector extracting step may include: calculating a plurality of Gabor size feature vectors expressed in polar coordinates in the normalized face image; And determining a face graph, which is a group of the Gabor size feature vectors, as a face feature vector.

The plurality of Gabor size feature vectors may be Gabor size feature vectors extracted from a grid structure position in the normalized face image.

The process of extracting the face graph as a face feature vector includes: a process of generating an inherent face cover space that is not related to illumination effects; Reducing a data size by projecting a Gabor size feature vector extracted at the grid structure position in the inherent face gaber space; And extracting a face graph, which is a group of Gabor size feature vectors projected in the inherent face gabber space, as a face feature vector.

The speech minutiae point extraction process may include a speech detection process for detecting speech in the entire audio using an average energy method and a zero crossing rate; And a voice feature data extraction process for extracting voice feature data from the extracted voice.

Detecting the voiced sound region in the entire audio through the average energy method; Determining whether there is a fricative at the front end of the voiced region using the zero crossing rate; And determining, as a speech, if there is a fricative sound in a front end of the voiced speech region.

 The voice feature data extraction process may include a fast Fourier transform (FFT) process of the extracted voice; Filtering the fast Fourier transformed speech data as a Mel filter which is a uniform filter; Discrete cosine transform (DCT) for eliminating the correlation of the filtered speech data; Extracting voice feature data by applying MFCC (Mixed Content Signal Classification) to the DCT voice data; And normalizing the number of the voice feature data for each frame applied to the MFCC.

The face similarity degree calculating process may calculate a correlation value between the registered feature vector registered in advance and the face feature vector extracted from the face image as the face similarity.

The voice similarity degree calculating step can calculate the correlation value between the registered feature data registered in advance and the voice specific data extracted from the voice as voice similarity.

Outputting a normalized face similarity normalized by a number within a range of 0 to 1, and outputting a normalized voice similarity normalized by a number within a range of 0 to 1; And applying an artificial neural network model for applying authentication to the ANN model by applying the normalized face similarity and the normalized voice similarity.

A process of learning an artificial neural network model composed of an input layer, a hidden layer, and an output layer by applying experimental data and a learning algorithm for inputting a desired output value together with an artificial neural network to learn; And performing authentication by applying the normalized face similarity and the normalized voice similarity to the artificial neural network model learned by the learning algorithm.

According to the embodiment of the present invention, stable user authentication can be performed by applying the data of face recognition and speech recognition to the neural network model. Therefore, according to the embodiment of the present invention, the error of the face recognition and the speech recognition can be reduced by using the artificial neural network.

1 is a flowchart illustrating an authentication process in which face recognition and speaker recognition are fused according to an embodiment of the present invention.
2 is a flowchart illustrating a facial feature point extraction process according to an embodiment of the present invention.
3 is a flowchart illustrating a speech minutiae point extraction process according to an embodiment of the present invention.
FIG. 4 is a diagram illustrating a discriminator of a tree structure connected in series according to an embodiment of the present invention, wherein a front discriminator detects a facial region using a simplified facial pattern. FIG.
FIG. 5 is a view normalizing the size of a detected face image according to an embodiment of the present invention by 256 × 256 pixels. FIG.
FIG. 6 is a diagram illustrating an edge and a valley in a slanted face region image according to an embodiment of the present invention; FIG.
FIG. 7 is a vertical histogram of an edge and a valley in an inclined face region image according to an embodiment of the present invention; FIG.
Figure 8 is an illustration of edges and valleys in an upright face region image in accordance with an embodiment of the present invention.
9 is a diagram illustrating edge and valley vertical histograms in an upright face region image in accordance with an embodiment of the present invention.
FIG. 10 is an angle normalized according to an embodiment of the present invention. FIG.
FIG. 11 is a view showing a state before and after removing a lighting effect according to an embodiment of the present invention; FIG.
12 illustrates a Gabor wavelet mask in accordance with an embodiment of the present invention.
FIG. 13 is a diagram of a garbage size feature vector extracted from a lattice structure position of a normalized facial image according to an embodiment of the present invention; FIG.
Figure 14 is a unique facial image illustration used to identify facial differences among others in accordance with an embodiment of the present invention.
FIG. 15 is an illumination diagram extracted from various lighting environments of the same person according to an embodiment of the present invention. FIG.
16 is a graph showing the relationship between Mel frequency and frequency according to an embodiment of the present invention.
17 illustrates clustering using K-means according to an embodiment of the present invention.
FIG. 18 is a view showing images used for face recognition according to an embodiment of the present invention; FIG.
FIG. 19 is a diagram showing an example of a waveform of a voice to be used in an experiment according to an embodiment of the present invention; FIG.
FIG. 20 is a diagram illustrating a result of similarity between speech and face according to an embodiment of the present invention; FIG.
21 is a diagram illustrating normalization of the similarity between face recognition and speech recognition according to an embodiment of the present invention.
22 is a diagram showing a result of similarity in speech recognition according to an embodiment of the present invention.
23 is a view showing a structure of an artificial neural network;
FIG. 24 is a diagram illustrating a structure of an artificial neural network for identifying an individual according to an embodiment of the present invention; FIG.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings, in order to explain the present invention in detail so that those skilled in the art can easily carry out the present invention. . Other objects, features, and operational advantages of the present invention, including its effects and advantages, will become more apparent from the description of the preferred embodiments. It should be noted that the same reference numerals are used to denote the same or similar components in the drawings.

FIG. 1 is a flow chart showing an authentication process in which a face recognition and a speaker recognition are fused according to an embodiment of the present invention. FIG. 2 is a flowchart illustrating a face feature point extraction process according to an embodiment of the present invention. 7 is a flowchart illustrating a speech minutiae point extraction process according to an embodiment of the present invention.

Biometrics techniques are used for various physiological and behavioral features such as fingerprint recognition, face recognition, speech recognition, and iris recognition. Fingerprint recognition is the most stable among these, and it has been widely used since it has been commercialized in many places.

The most important feature of biometrics is that it must be universal, unique for each person, and permanently unchanged. However, there is a disadvantage that the fingerprint is hard to recognize when it is worn out due to labor, when it is dry, or when foreign matter is buried. And most of all, research and development on other biometrics are underway because there is a feeling of refusal to use a machine that many people will recognize when they touch it.

Accordingly, the present invention proposes a stabilized authentication method with high security and reliability by applying the feature and similarity of face recognition and speaker recognition, which have no objection of the user, and which is convenient for the user, to the neural network model. An authentication method that combines face recognition and speaker recognition using an arithmetic device such as a CPU is presented.

In order to authenticate the fusion of face recognition and speaker recognition, a facial feature point extraction process (S100) for detecting a face image from the entire image acquired through a camera and extracting a face feature vector from the detected face image is performed.

An image used to recognize and detect a face is a mixture of face and background. Therefore, accurate face detection in the input full image must be preceded. In addition, facial images obtained under uncontrolled environment vary in size, posture, head shape, and chrominance distribution according to changes in distance, position, race, and illumination, so that a simple face brightness or intensity Or chrominance pattern matching is limited. Face detection using the currently known face color hue has the disadvantage of being sensitive to changes in the environment although it can detect faces with a small amount of computation.

In order to overcome this drawback, in the present invention, in order to accurately extract face regions of various sizes and orientations in a complex background in real time, a facial feature point extraction process (SlOO) includes a face image detection process (S110) A normalization process (S120), and a face feature vector extraction process (S130).

The face image detecting process (S110) is a process of detecting a face image step by step in the whole image. In order to detect the face image step by step, the face image is detected step by step through the adaboost method. The adaboost method is detailed in "An efficient illumination normalization method for face recognition" by Xudong Xie and Kin-Man Lam in the publication of "Pattern Recognition Letters".

For reference, the Adaboost method will be briefly described. In detecting a face image, the face area is detected using a stepwise method as shown in FIG. The face pattern used in each step of the stepwise detection uses a face pattern represented by an M × M (for example, 21 × 21) low resolution. This method is based on the fact that human cognitive systems search for faces even in low-resolution images. The number of face patterns that can appear in low-resolution images is still large (the number of patterns that can be expressed in a 21 × 21 binary image is "2 ⁴⁴¹ - 5.0 × 10 ¹³² "). In order to solve this problem, we simplified the face pattern to a simple basis pattern combination (about 100,000 combinations of Haar base patterns in 21 × 21 binary image) Learning method using a boosting algorithm in offline and face detection in real time on - line using a learned pattern has been successfully applied. A specific detection method detects faces in the background by serially connecting tree-classifiers of several tree structures as shown in [Figure 3-1].

As shown in FIG. 4, in the discriminator of the tree structure connected in series, the discriminator at the front end detects the facial region using the simplified facial pattern, and detects the facial region using a more complex facial pattern toward the rear end. Haar-based basis patterns consist of 2 ~ 4 rectangular blocks with given face area. The value of each rectangular block is the sum of all the pixel values in the rectangle and then weighted. The sum of the values of all the pixels in the rectangle can be calculated by about 10 addition operations using the sum image.

Upon completion of the face image detection process (S110), the normalization process (S120) normalizes the size and color of the detected face image to a set value.

As shown in FIG. 2, the normalization process (S120) includes a size normalization process (S121) of converting the size of the detected face image into a preset size and outputting the size as a normalized face image, An angle normalization process (S122) of obtaining a normalized face image by rotating the face image in a direction opposite to the inclination direction by detecting a tilting angle and a tilting direction, And an illumination normalization process (S123) for outputting the image as a normalized face image.

Since the size of the face image detected in the input image is not constant, a size normalization process (S121) is required to make the size of the detected face image constant to increase the efficiency of feature detection. That is, as shown in FIG. 5, the size of the detected face image is normalized to 256 × 256 pixels. In the following description, the size of the detected face image is assumed to be 256 x 256 pixels (pixels), for example.

After the size normalization process (S121) is completed, the tilted inclination angle and the inclination direction of the resized face image are grasped before the illumination normalization process, and the face image is rotated in the reverse direction of the inclination direction by the inclination angle An angle normalization process (S122) for outputting the size normalized face image may be further performed.

When the face is rotated in a plane in a normalized face image, the face is rotated to obtain a tilt angle and rotated by a tilting angle to straighten the face posture of the face image .

A vertical direction histogram calculation process, and a tilt angle determination process in order to grasp the tilt angle in the angle normalization process.

The vertical direction histogram calculation process is a process of calculating the vertical direction histogram in the valleys and edges in the vertical direction by extracting the valleys and edges in the vertical direction while rotating the size-converted face images at intervals of 2.5 degrees.

The inclination angle determination process is a process of determining the rotation angle when the vertical direction histogram calculated by rotating the resized face image at intervals of 2.5 degrees has the largest variance as the inclination angle.

In this case, among the resized face images, the brightness of the image pixel is lower than that of the surrounding pixels, and the brightness difference between the image pixel and the surrounding pixels is larger than a predetermined threshold value. The valley is defined as a portion having a brightness lower than that of the surrounding pixel, that is, a portion darker than the surrounding, while the brightness of the unknown pixel changes abruptly. In the present invention, a gray top-hat morphology operation is applied to a face region image, and then a binary closing morphology operation is applied to obtain a valley.

The edge refers to the face outline of the resized face image. Since the extraction of the valley and the edge is highly reliable since it is extracted within the correct face area, when the inner part of the face area, that is, the size of the detected face image is 256 × 256 pixels, the center 140 × 140 pixels ) -Level-sized region, the valley region and the edge are extracted. 6A and 6B, edges are shown in an inclined face image in FIG. 6, and valleys are shown in FIG. 6C.

6 (b) and Fig. 8 (b) show the edge, Fig. 6 (c) and Fig. 8 In the vertical direction. Comparing these results, it can be seen that the distribution of the histogram distribution in the vertical direction of the edge and valley (degree of change in the average) is larger than when the face is inclined when it is straight. Using this fact, the tilt angle (rotation angle with the greatest variance) was obtained and the face was straightened as shown in Fig.

After the normalization process (S121, S122) for size and angle is completed, an illumination normalization process (S123) is performed to remove illumination effect from the normalized face image and output it as an illumination normalized face image.

The facial image changes much even if it is the same person's face depending on the lighting environment during acquisition. Light can cause shadows near structures such as the nose, eyes, and mouth, or saturation can cause parts of the face to be poorly represented in the image. Therefore, it is essential to remove the influence of the illumination of the face before the authentication using the face feature. The deformation of the face image due to the illumination effect is a deformation caused by adding a multiplicative moise A and an additive noise B to the image I (x, y) excluding the illumination effect as shown in the following equation.

I '(x, y) = A x I (x, y) + B

Here, I '(x, y) is the illuminated face image

I (x, y): illuminated normalized facial image

Eliminating this illumination effect is to remove multiplicative moises and additive noise in the normalized face image. The present invention eliminates multiplicative moise and additive noise by the following equation (1) without directly calculating multiplicative moises and additive noise. have.

(X, y) is the pixel function of the illuminated face image, I (x, y) is the pixel function of the illuminated normalized face image, E is the normalized When the average of the pixel values of the facial image, VAR,

[Equation 1]

The illumination effect can be removed.

11 (a) shows an image image before the illumination effect is removed, and FIG. 11 (b) shows an image image after the illumination effect is removed.

Meanwhile, when the normalization process (S120; S121, S122, and S123) for size, angle, and illumination is completed, a face feature vector extraction process (S130) for extracting a face feature vector from the normalized face image is performed.

The face feature vector extraction process includes a step S131 of calculating a plurality of Gabor size feature vectors expressed in polar coordinates in a normalized face image, a step S132 of determining a face graph as a group of Gabor size feature vectors as a face feature vector ).

In the present invention, a Gabor size feature vector is used as a face feature vector. The Gabor size feature vector is obtained by taking only the magnitude value from the Gabor feature vector represented by polar coordinates. The Gabor size feature vector at the face image feature points is defined as a set of Gabor coefficients obtained by convolution with different Gabor wavelet kernels constructed according to direction / frequency / phase for facial image feature points. The Gabor wavelet kernel used in the present invention is described in "Wiskott, JM Fellous, N. Kuiger, C. von der Malsburg," Face Recognition by Elastic Bunch Graph Matching, "Pattern Analysis and Machine Intelligence, IEEE Transactions, Vol. -779, July, 1997. "is used.

That is, the Gabor wavelet kernel used in the present invention can be expressed as [Equation 2] as follows.

&Quot; (2) "

이고 웨이브 벡터

는

로 주어지며, 이 때

는 웨이블렛의 방향을

는 웨이블렛의 파장(주파수 역수에 비례)을 나타낸다. 또한

는

에 비례하는 가우시안의 크기를 나타낸다. 본 발명에서는 가버 웨이블렛 커널에 대해

와

,

의 40개 조합으로 나타나는 가버 웨이블렛 커널을 사용하였다. 본 발명에서는 실제적으로 가버 특징 벡터들은 다음과 같이 구하였다. 상기 40개 조합에 대한 가버 웨이블렛 커널을 실수부와 허수부로 나누고 각각을 이산화 하여 가버 웨이블렛 마스크들을 만들고 이 j 번째 조합에 대한 실수부 및 허수부 가버 웨이블렛 마스크들과 이미지의 점

근방 각 점에서의 이미지 픽셀값(그레이값)들과 컨볼루션하여 j 번째 가버 특징 벡터

(여기서

)을 구하였다.here,

And the wave vector

The

Lt; / RTI >

The direction of the wavelet

(Proportional to the frequency inverse number) of the wavelet. Also

The

The size of Gaussian proportional to In the present invention, the Gabor wavelet kernel

Wow

,

The Gabor wavelet kernel is used. In the present invention, the Gabor feature vectors are actually obtained as follows. The Gabor wavelet kernels for the 40 combinations are divided into a real part and an imaginary part and are respectively discretized into Gabor wavelet masks, and real and imaginary Gabor wavelet masks for this j-th combination and points

(Gray values) at the neighboring points to obtain a j-th Gabor feature vector

(here

) Were obtained.

에서의 가버 특징벡터

은

으로 정의된다. 또한 각 복소 가버 웨이블렛 계수

은

(크기

, 위상

)로 표현될 수 있다. 이때 가버특징벡터

에서 그 크기값 많을 가져와 만든 벡터 a_j , j=1,....40를 가버 크기 특징 벡터라 한다. 참고로 도 12는 본 발명의 실시예에서 사용한 80개의 가버 웨이블렛 마스크(40개 조합 가버웨이블렛 커널

2(실수부, 허수부))를 나타낸다.At this time,

The Gabor feature vector

silver

. Each complex Gabor wavelet coefficient

silver

(size

, Phase

). ≪ / RTI > At this time,

The vector a _j , j = 1,. For reference, FIG. 12 is a schematic diagram of an embodiment of the present invention, in which 80 Gabor wavelet masks (40 combination Gabor wavelet kernels

2 (real part, imaginary part)).

In addition, the Gabor size feature vectors are Gabor size feature vectors extracted at the grid position in the normalized face image.

A face graph was formed by Gabor feature vector obtained by 2D convolution with Gabor wavelet at feature point such as nose tip and mouth tail and used as facial feature information. However, this method requires a large amount of computation to find the feature point, and it is difficult to detect the exact feature point position when the initial feature point estimation position is away from the original position. Therefore, in the present invention, a Gabor size feature vector is extracted from a grid structure position of a normalized face image as shown in FIG. 13, and a face graph of a group of Gabor size feature vectors is defined as a face feature vector. This method is known to have no significant difference in performance from the facial feature vector extracted from the biometric feature points.

After calculating a plurality of Gabor size feature vectors expressed in polar coordinates in the normalized face image (S131), a face graph, which is a group of Gabor size feature vectors, is determined as a face feature vector (S132).

The process of making the face graph as a face feature vector (S132) includes a process of generating an inherent face garbage space that is independent of the lighting effect, a step of projecting the garbage face feature vector extracted from the grid structure position in the inherent face garbage space, And a face graph, which is a group of Gabor face feature vectors projected on the inherent face gauge space graph, as a face feature vector.

The extracted facial feature vector is a collection of Gabor size feature vectors of 40 lengths extracted from 192 lattice points and each dimension value of the Gabor size feature vector is a 4 byte float value. Since the large size information of 30K bytes can not be handled by the smart card, it is necessary to reduce the size of the face feature vector while minimizing the information loss.

To this end, in the present invention, a native face garber PCA space is generated through an independent natural face garber space, for example, a PCA analysis, and a face characteristic vector is projected in a native face garber space without minimizing illumination. Hereinafter, the inherent face gabber space will be described as an example of the inherent face gaver PCA space through PCA analysis. The inherent face gabber space is obtained by the fact that the face image is composed of illumination image and reflection image, and that the PCA analysis represents the distribution characteristic of the data best and the principal component axis which minimizes the loss of information at projection. The face image I (x, y) is a reflection image R (x, y) reflecting the intrinsic characteristics of the face and the illumination image L (x, y) .

&Quot; (3) "

I (x, y) = R (x, y) L (x, y)

However, the reflection image R (x, y) reflecting the intrinsic characteristics of the face is not independent of the illumination image L (x, y). The two come from the face image I (x, y) and are related to each other by I (x, y). Therefore, the space constituted by the facial feature vectors obtained from the reflection image R (x, y) is also not orthogonal to the space constituted by the facial feature vectors obtained from the illumination image L (x, y).

However, it can be said that the facial feature vector extracted from the normalized facial image using the illumination normalization technique and the facial feature vector extracted from the illumination image still reflect the characteristics of the facial feature and the reflection effect, respectively .

In addition, assuming that most of the faces of a person are similar, and most of the effects of lighting also occur in the face, the PCA space can be generated as follows.

For this purpose, robust facial recognition based on the PCA model of feature vector illumination, which explains the Garber PCA space through PCA analysis, "Sanghoon Kim, Seoltae Jung, Jeon Sun Tae, Cho Sung Won, Gabor, SC 45 (6), pp. 67-76, 2008 . In accordance with the present invention, a facial feature vector is extracted from each person's facial image (FIG. 14) normalized to obtain a unique facial feature PCA space, and a PCA is performed on the facial feature vector data set. Let Φ _R be the matrix with the principal vector obtained by this process as a column vector. The PCA is performed on the face feature vector extracted from the illumination image of the same person (FIG. 15) taken in various illumination environments. And if it is called a matrix having a principal component vector obtained as a column vector by Φ _L, principal component axis in Φ _R are a base that best represents the face difference in many people, as the main component axes are effective for lighting in Φ _L It is the basis for expressing the changing facial features. However, assuming that most people's faces are similar and the effect of illumination is almost similar to that of all people, Φ _L is not a model of lighting effect for a specific person, but rather a general illumination The base Φ ^* _{R of} the space orthogonal to the change of the illumination effect can be obtained as the _following Equation (4).

&Quot; (4) "

That is, by removing the components that are not orthogonal to each other from the bases Φ _R and Φ _L of the spaces that are not orthogonal to each other, a new base Φ ^* _R orthogonal to the illumination effect can be obtained while expressing the unique characteristics of the face best.

Finally, it is possible to achieve information compression that minimizes information loss by projecting the facial feature vector into the newly constructed unique facial garbage PCA space. In the actual experiment, the 292-basis facial feature vectors of 292 dimensions were obtained by generating the native face garber PCA space of 292 bases. In addition, since the PCA space is independent of the illumination, it can be robust against illumination.

Meanwhile, in addition to the facial feature point extraction process (S100), a speech feature point extraction process (S200) is performed to detect speech from audio acquired through a microphone and extract speech feature data from the detected speech.

The voice feature point extraction process (S200) has a voice detection process (S210) and a voice feature data extraction process (S220) as shown in FIG.

The voice detection process (S210) detects the voice in the entire audio using the average energy method and the zero crossing rate. That is, the voice detection process (S210) includes a process S211 for detecting a voiced region in the entire audio through the average energy method, a process S212 for determining whether a fricative exists in the front of the voiced region using the zero crossing rate, And a step S213 of judging the presence of a fricative sound in the front end of the voiced sound region as speech.

In order to extract a speech feature from a speech signal for speaker recognition, it is necessary to first extract a portion having a real speech. For this, a short time analysis of the average energy method and zero crossing rate is used.

The energy of the speech signal changes with time. Short-time energy is a voice parameter that can effectively represent the amount of energy change. Especially, in the case of voiced sound, because it has a larger energy component than unvoiced sound or noise, the average energy method is used to extract the voiced sound part. The equation for obtaining the average energy is shown in Equation (5) below.

&Quot; (5) "

Here, N means the number of samples in one frame, and w (n) is a window function for setting the corresponding interval.

Also, the zero crossing rate is a measure of how often the sign of two consecutive samples in a speech signal is different, ie how frequently the value of the sampled speech signal crosses zero. The portion to be pronounced, that is, the portion considered as a voice, does not mean only a voiced sound. Since the unvoiced part is small in energy, it can not be extracted by the average energy method alone. Therefore, in the case of unvoiced sound, it can be separated by considering that the frequency component is larger than voiced sound and general noise. When the frequency of the voice signal is large, the zero crossing rate is large, and when the voice signal is small, the zero crossing rate is small. Therefore, in the case of unvoiced sound, since the zero crossing rate is larger than that of voiced sound, it is used to extract the unvoiced sound portion, and the formula for obtaining the zero crossing rate is as shown in Equation (6).

&Quot; (6) "

If the measured zero crossing rate is greater than the determined threshold, it can be seen that the unvoiced sound exists in the corresponding part. Therefore, in order to extract the speech part, first the voiced part is searched by the average energy method and then the voiced part is found through the zero crossing rate.

On the other hand, when the voice detection process (S210) is completed, the voice feature data extraction process (S220) extracts voice feature data from the extracted voice as shown in FIG. The voice feature data extraction process includes a process of performing a fast Fourier transform (FFT) process S221, a process of filtering as a Mel filter S222, a DCT transform process S223, a mixed content signal classification (MFCC) (S224) of extracting voice feature data, and a process of normalizing the number of voice feature data (S225).

Describing the speech feature data extraction process, first, there is a fast Fourier transform (FFT) process (S221) of the extracted speech.

에 처리할 수 있다. DFT 계산을 FFT로 대치하면

의 연산량 감소율을 얻을 수 있다. 만약 N이 128이라면 약 18배, N이 256이라면 약 32배의 연산량 감소 효과를 갖는다. 고속 퓨리에 변환(FFT)의 원리는 다음과 같다. 만약 푸리에 변환을 알고자 하는 신호의 샘플개수 N이 2의 m승이라고 하면, N개의 샘플신호를 [수학식 7]과 같이 차례로 홀수 번째 샘플열과 짝수 번째 샘플열의 2개의 샘플 열로 분류할 수 있다.In order to obtain a discrete Fourier transform (DFT) spectrum from N discrete signals on the time axis, a computation amount of N squared is required. This is because it requires N sums and complex multiplications for every N elements of the DFT. If N is 128, the sum of 16384 times and the complex number must be multiplied. If N is 256, the sum of 65536 times and the complex number must be multiplied. Therefore, it is difficult to process DFT in real time, which requires a computation amount of N squared. Fast Fourier Transform (FFT) is an algorithm developed to solve this computation problem. Using Fast Fourier Transform (FFT), the computation of N squared?

Lt; / RTI > Replacing the DFT calculation with an FFT

The reduction rate of the calculation amount of the operation amount can be obtained. If N is 128, it is about 18 times. If N is 256, it is about 32 times smaller. The principle of fast Fourier transform (FFT) is as follows. Assuming that the number N of samples of the signal for which the Fourier transform is to be known is an m-th power of 2, the N sample signals can be classified into two sample sequences of an odd-numbered sample sequence and an even-numbered sample sequence, as shown in Equation (7).

&Quot; (7) "

The two sample sequences g and h each have N / 2 samples and have a period of NT / 2.

는 [수학식 8]과 같이 정의된다.After the extracted voice is subjected to fast Fourier transform (FFT), there is a step S222 of filtering the voice data subjected to the fast Fourier transform (FFT) as a Mel filter which is a uniform filter. Since there are too many data values per frame, we get the energy of each filter bank through the filter bank with the result of FFT to extract the appropriate parameters. The most common form of filter bank used in speech recognition is a uniform filter bank. Here, the center frequency value of the i-th band pass filter,

Is defined as " (8) "

&Quot; (8) "

In Equation (8), Fs is the sampling frequency of the speech signal, and N is the number of uniform distribution filters dividing the sampling frequency by equal intervals. Also, Q representing the number of actual filters satisfies the following condition.

&Quot; (9) "

Q? N / 2

In Equation (9), the equal sign is established when the entire frequency range of the speech signal is used for analysis. The band b _i of the i-th filter generally satisfies the following equation (10).

&Quot; (10) "

b _i ≥ F _s / N

The equal sign means that there is no frequency overlap with adjacent filter channels. Inequality signifies that adjacent filter channels overlap. If b _i <F _s / N, the information of a part of the speech spectrum loses its meaning and correct analysis can not be performed.

The mel filter used in the embodiment of the present invention may be a Mel Filter Bank in which a plurality of mel filters are combined. The Mel filter bank is a nonuniform filter bank. When designing a nonuniform filter bank, there is a method of directly using the main band unit. The placement of filters along the main band is based on cognitive research. The spacing of these filters is linear at frequencies below 1000 Hz and closer to logarithmic units, as shown in FIG. In the embodiment of the present invention, the mel-frequency region is shifted to the Mel frequency region using the Mel formula of Equation (11), and then the region is equally divided in the Mel frequency region and then converted to the frequency domain.

&Quot; (11) &quot;

And a discrete cosine transform (DCT) process (S223) for eliminating the correlation of the filtered voice data after filtering as a Mel filter. The DCT removes the correlation between the outputs from the filter banks and collects the characteristics of the parameters.

After the discrete cosine transform (DCT), the speech characteristic data is extracted by applying MFCC (Mixed Content Signal Classification) to the discrete cosine transform (DCT) speech data (S224). When MFCC (Mixed Content Signal Classification) is applied to one frame, the feature vector value is outputted by the number of filter banks. In the feature extraction process, the number of filter banks is selected from a small degree to a required number in the feature order. The process for obtaining the MFCC is expressed by the following equation (12).

&Quot; (12) &quot;

In the above, y _t ^(m) is the output obtained from the bank of melfilter, and the feature vector y _t ^(m) (k) can be obtained by taking the logarithm and performing the DCT.

In addition, after extracting the voice feature data by applying the MFCC (S224), the process of normalizing the number of the voice feature data for each frame applied to the MFCC is performed (S225). In case of voice through MFCC, 13 feature vectors can be obtained in one frame. However, since the number of frames is more than 120, and the number of frames is less than 60, the number of frames can not be compared with each other. In the present invention, the K-means (K-means) algorithm is used to group the respective voice information data obtained through the MFCC into one normalized data as shown in FIG. Thus, in the embodiment of the present invention, the data is clustered into 13 × 20 through the K-means algorithm. K-means is one of clustering techniques. Clustering is mainly used for segmentation, and it is used for separating large data into small data groups.

For reference, the K-means algorithm is the most commonly used partitioned clustering algorithm. The following Equation (13) briefly shows the K-means algorithm.

&Quot; (13) &quot;

라 하고 하기의 [수학식 14]와 같이 정의된다.The concept of this K-means algorithm is to minimize the average Euclidean distance between the patterns and the center of the cluster to which the pattern belongs. The center of the cluster is the average or centroid of the patterns in the cluster.

And is defined by the following equation (14).

&Quot; (14) &quot;

In Equation (14), w is a pattern set belonging to a cluster, and x is a specific pattern belonging to a cluster. The pattern is represented by a vector with real values. In K-Means, clusters are thought of as spheres with center of gravity, like the center of gravity. The RSS (Residual Sum of Squares) indicating how well the center represents the patterns belonging to the cluster is represented by the sum of squares of the respective patterns and the center to all the patterns belonging to each cluster, ]. RSS is the objective function of the K-means and should be minimized.

&Quot; (15) &quot;

Meanwhile, after extracting the facial feature points and audio feature points is calculated (S300) a face similarity between the facial feature vector extracted from the pre-registered registered feature vectors, and a face image.

The face similarity degree calculation (S300) calculates the correlation value between the registered feature vector registered in advance and the face feature vector extracted from the face image as the face similarity. The registration feature vector refers to a feature vector extracted from the original face registered in advance, and the face feature vector refers to a feature vector extracted from the face of the captured image for authentication.

The similarity between two facial feature vectors is defined as the correlation value between two facial feature vectors normalized to 1 in size. If the similarity is obtained more than the experimentally obtained threshold value, the method is regarded as the same person, otherwise, .

&Quot; (16) &quot;

: 등록자의 얼굴 특징 벡터

: Registrant's facial feature vector

: 피인증자의 얼굴 특징 벡터

: The facial feature vector of the subject

In the embodiment of the present invention, an 8-megapixel Galaxy Note 2 built-in camera, a LifeCamVX-7000 webcam, and a built-in camera of a 5G iPhone camera are used to create a face image. In order to avoid the influence of posture and lighting environment on the images of the face used in the experiment, the experiment was conducted with two frontal faces of 50 persons and a total of 100 images taken in the same lighting environment. 18 are images used for face recognition.

The similarity is calculated by comparing the facial feature vector extracted from each facial image and the face feature vector extracted from one face image of the same person and the face feature vector extracted from 50 different face images, And judge whether or not they are the same person. Table 1 summarizes the experimental results of face feature vector matching for various thresholds.

Threshold FAR
(False Accept Rate) FRR
(False Reject Rate)
0.4

21% (21/100 times)
4% (4/100 times)
0.5

9% (9/100 times)
4% (4/100 times)
0.6

2% (2/100 times)

6% (6/100 times)
0.7

0% (0/100 times)
10% (6/100 times)
0.8

0% (0/100 times)
14% (14/100 times)

Experimental results show that when the threshold value is 0.7 or more, the FRR value has a relatively low FAR value, but when the threshold value is set to 0.7 or less, the FRR value is lowered, but the FAR value is increased have.

A face similarity degree calculation step (S300), and a voice similarity degree calculation step (S400) of calculating a voice similarity degree between the registration feature data registered in advance and the voice specific data extracted from the voice. The voice similarity degree calculation step (S400) calculates the correlation value between the registered characteristic data registered in advance and the voice specific data extracted from the voice as the voice similarity degree. Herein, the registration feature data refers to feature data extracted from the original voice registered in advance, and the voice feature data refers to feature data extracted from the audio voice for authentication.

The characteristics of speech recognition are regarded as the same person if the correlation value that is obtained through the experimentally obtained threshold is obtained by applying the correlation method used in the face recognition to equalize the input parameters of the artificial neural network, .

In the embodiment of the present invention, the MP5000 speech-only microphone of ABKO was used to obtain the feature data of the speech recognition. Experiments were carried out by using the same vocabulary as 10 vocabularies and 20 vocabulary files for each vocabulary. 19 is an example of a waveform of a voice to be used in an experiment.

In this way, the speech feature vector is extracted through the waveform of each speech, and two speech of the same person is experimented to obtain the FRR. In order to obtain the FAR, it alternates with the remaining nine voices based on the same voice of each person, and compares with the voice feature vector to determine whether the person is the same person or not. Table 2 below is a table showing results of matching experiments of speech feature vectors.

Threshold FAR
(False Accept Rate) FRR
(False Reject Rate)
0.65

34.4% (124/360 times)
205 (4/20 times)
0.7

31.1% (112/360 times)
30% (6/20 times)

Experimental results show that the values of FAR and FRR change according to the value of the threshold. However, it shows higher error rate than face matching result.

Meanwhile, after the face similarity degree and the voice similarity degree are calculated (S300 and S400), the user authentication process (S500) is performed to apply the calculated face similarity and voice similarity to the ANN model to perform the identity authentication.

And to find and predict the artificial neural network which can obtain the best recognition result by converging the data of the matching information with the face and the voice of the user. The simplest way to do this is to evaluate the error and recognition rate using different information data than the data used in the learning. Therefore, the efficiency of the selected artificial neural network can be verified by learning the artificial neural network to minimize the error by train data and measuring the efficiency of new data of test data.

In the embodiment of the present invention, a similarity between face recognition and speech recognition is compared and applied to an artificial neural network model, thereby realizing a system in which identity verification can be predicted.

Before applying the face similarity and the voice similarity to the artificial neural network model, the normalized face similarity normalized by the numbers in the range of 0 to 1 is output first, and the normalized voice similarity normalized by the numbers within the range of 0 to 1 (S510). The input variables to be applied to the artificial neural network are switched to have the same range of values. Since the units of a given input variable are different. The data suitable for the neural network model should be categorical input variables having a frequency more than a certain frequency in all categories, and the range of continuous input variable values should not be much different among the variables.

In the embodiment of the present invention, the degree of similarity of face recognition has a value of 10 ⁹ and the degree of similarity of speaker recognition has a value between 0 and 1. Since the magnitudes and ranges of these numbers are very different, we normalize each input variable to start learning at an equivalent level and determine the weights. 20 shows a normalized result of face recognition and speaker recognition similarity. As shown in FIG. 20 showing the result of similarity between voice and face, normalization was required because the range and distribution of input values were too large. As shown in FIG. 21, these values are normalized so that the range and distribution of values are not dispersed. The reason for normalizing this to one digit is to match the distribution of size or range equally, but it is a result of matching between face recognition and speaker recognition, and it shows wrong result in confirmation of identity due to slight difference.

22 shows the result of matching 3.txt and 4.txt and 15.txt and 16.txt for the voice of the same person. However, if the threshold value is set to 0.7, it is judged that the person is not the person, and normalization of the size is performed to make the voice of the person himself.

After the normalized face similarity and the normalized voice similarity are calculated as described above, the authentication of the identity is performed by applying the normalized face similarity and the normalized voice similarity to the ANN model.

Artificial neural networks are briefly described in detail. Artificial neural network is a network constructed by mimicking human brain and neuron model and is known to have excellent learning ability and reasoning ability. This model is a parallel interconnected network of simple constituent elements and their hierarchical organization, just as biological objects interact with real world objects like biological neural systems. In other words, the connection state and the connection strength between the processing unit and neuron's transition function and the number of layers indicating the structure of the neural network are adjusted to suit a given problem.

The structure of an artificial neural network consists of an input layer, a hidden layer, and an output layer. These are the layers of processing elements together. The processing element can be called an activation function, which consists of two parts. First, the first part accepts input from several different processing elements and computes the net input value using the connection weight. This is done by a combination function. The second activation step is performed by the transition function, which converts the input value taking the weight into an output value using a transfer function. The functions that can be considered as transition functions are linear functions, Sigmoid functions, and hyperbolic tangent functions. Linear functions are similar to those used in linear regression and the latter are nonlinear functions and the results are nonlinear.

In the artificial neural network, processing elements are gathered to form a layer. FIG. 23 shows the structure of the artificial neural network. In Figure 23, the input layer is the layer in which the data is provided in the neural network, and the output layer is the layer in which the artificial neural network responds to the given input. The hidden layer exists between the input layer and the output layer and is completely connected to all nodes of the input layer. The nodes in the hidden layer calculate the value by multiplying each input by the corresponding weight, and pass it to the output layer.

In the embodiment of the present invention, a learning algorithm for performing authentication by applying normalized face similarity and normalized voice similarity to an artificial neural network model (S520) by inputting experiment data and a desired output value into an artificial neural network Propagation learning algorithm) is applied to learn an artificial neural network model composed of an input layer, a hidden layer, and an output layer. Then, by performing the identity authentication by applying the normalized face similarity degree and the normalized voice similarity degree to the artificial neural network model learned by the learning algorithm, error-free identity authentication can be performed.

The learning algorithm used in the artificial neural network used in the present invention is a known back propagation learning algorithm. The back propagation algorithm compensates the disadvantages by extending the ADALINE (Adaptive Linear Neuron) method in multiple layers. For reference, ADALINE (Adaptive Linear Neuron) is an initial model for an artificial neural network neuron, in which an adaptive linear combiner and a quantization circuit are connected in series. When ADALINE is compared with a neuron, the adaptive weight corresponds to the synapse, the input vector component corresponds to the input of the axon projection, and the quantized axial force corresponds to the axon output. The output of ADALINE is very similar to what happens in real neurons.

The basic principle of the backpropagation learning algorithm is as follows. Each neuron in the input layer is given an input pattern, which is transformed at each neuron and delivered to the middle layer, and finally at the output layer. The output value is compared with the expected value to adjust the weight in the direction of reducing the difference, and the weight is adjusted based on the backward propagation in the upper layer and the lower layer. In map learning, input and desired output (target output) patterns (vectors) are indicated in artificial neural networks. Learning of the artificial neural network using the backpropagation learning algorithm consists of three steps. The learning input pattern is input to the artificial neural network to obtain the output. In the second step, the difference between the output and the target value, that is, the error is obtained. If there is no error value in the third step, no learning occurs. The weight of the output layer and the weight of the hidden layer are changed in the direction of reducing the magnitude of the error value while propagating the value in the opposite direction. Although the backpropagation learning algorithm is easy to misunderstand that the backpropagation learning algorithm is an artificial neural network with a circular structure, only the output related to the error is propagated in the reverse direction only in the learning process and the learning is completed. In actual application, It is necessary to make clear that it is a forward artificial neural network structure. In addition, the backpropagation learning algorithm takes a considerable amount of time to learn, but once the learning ends, it outputs the results very quickly at the application level.

 To further elaborate on the backpropagation algorithm, the backpropagation learning algorithm changes the weights between the hidden layer and the output layer using the output layer error signal as in the general delta rule learning method, To change the weights between the input layer and the hidden layer.

First, after selecting the learning p number of training patterns to a pair (x1, s1), (x2 , s2) ,,,,, (x p, s p), v the initial weight, and initializes the value of w to an arbitrary small , And determines an appropriate learning rate? Based on the learning rate selection method. Learning pattern pairs are input in order, and weights are changed while learning. The backpropagation learning algorithm uses the sigmoid function as the active function. The square error E is obtained by comparing the target value s and the final output y as shown in the following equation (17).

&Quot; (17) &quot;

The error signal? _Y of the output layer is obtained by the following equation (18).

&Quot; (18) &quot;

The error signal? ₂ propagated to the hidden layer is obtained by the following equation (19).

&Quot; (19) &quot;

및 입력층과 은닉층간의 가중치 변화량

를 구한다.Further, the weight variation between the hidden layer and the output layer in the k learning step

And a weight change amount between the input layer and the hidden layer

.

&Quot; (20) &quot;

, 입력층과 은닉층간의 가중치

을 구한다.Also, by using the following expression (21), the weight of the hidden layer and the output layer in the (k + 1)

, The weight between the input layer and the hidden layer

.

&Quot; (21) &quot;

The learning pattern pair is input repeatedly to change the connection strength. When the error is less than the specified range, the learning is terminated.

Accordingly, in the present invention, a learning algorithm (backpropagation learning algorithm) is applied to perform authentication by applying normalized face similarity and normalized voice similarity to an artificial neural network model by inputting experimental data and desired output values into an artificial neural network Learning of an artificial neural network model composed of an input layer, a hidden layer, and an output layer also follows the known artificial neural network learning algorithm described above.

In other words, the backpropagation algorithm used in artificial neural network model for identity confirmation prediction is a learning algorithm used in multi-layer and feedforward neural network, and the learning method is supervised learning. In the present invention, as shown in FIG. 24, a three-layer structure including an input / output layer and an intermediate layer is used, and input layers use respective feature information.

As shown in FIG. 24, the number of input layers is two for the comparison of similarity of face and voice, and the number of hidden layers is six. The output layer is set to two, and the learning is performed so that the first node is the first node if the node is the first node and the second node if the node is not the node. The back propagation algorithm used in the artificial neural network model for identification requires a learning process to adjust the weight so that the error between the desired output and the actual output value of the artificial neural network is minimized for a given input. As described above, the algorithm of the artificial neural network model provided in the embodiment of the present invention is composed of initialization of a weight vector, updating of a weight vector, and generation of a new weight vector. The learning algorithm is as follows.

STEP1. Feedforward propagation through a communication network

j = 1,2 ,,,, 6

STEP2. Error calculation in output layer

δ _k = designed output, k = 1,2 ,,, 10

STEP 3. Backward propagation of error in middle layer

STEP4. Update weights

α = learning rate

STEP5. Repeat steps 1-4 until convergence

는 0.1로 하였다. 학습률이 높으면 빠른 학습이 가능하지만, 수렴하지 않고 오답을 낼 확률 또한 커진다.
As you can see from the learning algorithm, it will calculate the error value and update the weight with continuous iteration. In this study, the number of iterations is set to 2000000, and the threshold for the total error is set. When the threshold is reached during the iteration, the learning is completed. The threshold was set to 0.01

Lt; / RTI &gt; If the learning rate is high, fast learning is possible, but the probability of giving wrong answers without convergence also increases.

On the other hand, after the artificial neural network is learned, normal authentication face similarity and normalized voice similarity are applied to the artificial neural network model learned by the learning algorithm (S520), thereby performing identity authentication without error.

Table 3 below shows experimental results obtained by learning one data of the data a and b including the subject's face and voice and the remaining 9 data in the artificial neural network, and then testing the data of the other person. [Table 4] shows the result of testing two different data of my own after learning the data of the other nine without the data of my own.

False Accept Rate (FAR) False Reject Rate (FRR) A 0/36 0/2 B 0/36 0/2 C 0/36 0/2 D 0/36 0/2 E 0/36 0/2 F 1/36 0/2 G 0/36 0/2 H 1/36 0/2 I 0/36 0/2 J 0/36 0/2 result 2/360 0/20

False Accept Rate (FAR) False Reject Rate (FRR) A 0/36 0/2 B 0/36 0/2 C 0/36 0/2 D 0/36 0/2 E 1/36 0/2 F 1/36 0/2 G 0/36 0/2 H 1/36 0/2 I 0/36 0/2 J 0/36 2/2 result 3/360 2/20

If we input the similarity value of face and speaker in artificial neural network, we could not get accurate results. As a result of testing after normalizing the values of each input data, the result of learning the subject showed 100% of the recognition rate of the person, but when the test was performed without learning the person, the recognition rate was 90% .

During experiments that did not teach me, FRR showed both errors at 'J'. The matching result in the face image is higher than the matching result in the voice image, but in the case of 'j', the matching result in the voice is higher than the matching result in the face image, Respectively.

False Accept Rate (FAR)

False Reject Rate (FRR)
Result 1

0.56%
0%
Result 2

0.83%
10%

Referring to Table 5 above, when the result of fusion of face and voice is compared with the result of matching of face and voice, excellent results are obtained than the result of voice matching. However, when the data including the face and the voice of the user were compared with the results of the face matching, the results were better than the results of the face matching. However, when the user did not include the user, The results are similar to those of the results.

As can be seen from the above results, incorrect distribution of values due to improper authentication of speaker authentication resulted in incorrect recognition result such as j. If the speaker authentication result is improved, the result of fusion authentication will be stable and accurate.

The embodiments of the present invention described above are selected and presented in order to assist those of ordinary skill in the art from among various possible examples. The technical idea of the present invention is not necessarily limited to or limited to these embodiments And various changes, modifications, and equivalents may be made without departing from the spirit and scope of the present invention.

S100: Facial feature point extraction S200: Speech feature point extraction
S300: calculation of face similarity degree S400: calculation of voice similarity degree
S500: Self certification process
S510: Normalized face similarity, normalized voice similarity calculation
S520: Application of artificial neural network model

Claims (19)

A face feature point extraction process for extracting a face feature vector from a detected face image by detecting a face image from the whole image;
A voice feature point extracting step of detecting voice in audio and extracting voice feature data from the detected voice;
A face similarity degree calculating step of calculating a face similarity degree between a registered feature vector registered in advance and a face feature vector extracted from the face image;
A voice similarity degree calculating step of calculating a voice similarity degree between pre-registered registered characteristic data and voice specific data extracted from the voice; And
A person authentication process for applying the face similarity and voice similarity to an artificial neural network model to perform identity authentication;
Wherein the face recognition and the speaker recognition are combined.
The method according to claim 1,
A face image detecting process for detecting a face image step by step in a whole image;
A normalization process of normalizing the size and color of the detected face image to a set value; And
A face feature vector extracting step of extracting a face feature vector from the normalized face image;
Wherein the face recognition and the speaker recognition are combined.
The method according to claim 2, wherein stepwise detection of the face image comprises:
An authentication method that is a fusion of face recognition and speaker recognition that detects face images step by step through an adaboost method.
The method of claim 2,
A size normalization process of converting the size of the detected face image into a predetermined size and outputting the size normalized face image;
Wherein the face recognition and the speaker recognition are combined.
5. The method of claim 4, wherein after the size normalization process is completed,
And an angle normalization process of grasping an inclined angle and an inclination direction of the size-converted face image, rotating the face image in a direction opposite to the inclination direction by the inclination angle, and outputting the rotated face image as a size normalized face image An authentication method that combines face recognition and speaker recognition.
[6] The method of claim 5, wherein, in the angle normalization process,
Extracting valleys and edges in the vertical direction while rotating the size-converted face images at intervals of 2.5 degrees, and calculating a vertical direction histogram in valleys and edges in the vertical direction; And
And determining a rotation angle when the vertical direction histogram calculated by rotating the resized face image at intervals of 2.5 degrees has the largest variance as an inclination angle,
The valley is a region in which the brightness of the image pixel in the resized face image is lower than that of the surrounding pixels and the brightness difference between the image pixel and the surrounding pixels is larger than a predetermined threshold value, And a face contour of the face image. The authentication method is a fusion of face recognition and speaker recognition.
The method of claim 5, wherein after the angle normalization process,
And an illumination normalization process of removing the illumination effect from the normalized face image and outputting the illumination normalized face image as an illumination normalized face image.
8. The method of claim 7,
And a face recognition and a speaker recognition which are output as an illumination normalized face image by eliminating multiplicative moise and additive noise from the normalized face image.
The method of claim 8, wherein removing the multiplicative moise and additive noise in the normalized face image comprises:
(x, y) is the pixel function of the illuminated face image, I (x, y) is the pixel function of the illuminated normalized face image, E is the normalized face When the average of pixel values of an image and VAR are variance,

And the face recognition and the speaker recognition are fused.
The method according to claim 2,
Calculating a plurality of Gabor size feature vectors expressed in polar coordinates in the normalized face image; And
Determining a face graph, which is a group of the Gabor size feature vectors, as a face feature vector;
Wherein the face recognition and the speaker recognition are combined.
11. The method of claim 10,
Wherein the normalized facial image has garbage size feature vectors extracted at a grid structure position in the normalized facial image.
The method according to claim 11, wherein the step of extracting the face graph as a face feature vector comprises:
A process of generating an inherent face garber space that is not related to lighting effects;
Reducing a data size by projecting a Gabor size feature vector extracted at the grid structure position in the inherent face gaber space; And
Extracting a face graph, which is a group of Gabor size feature vectors projected in the inherent face gabber space, as a face feature vector;
Wherein the face recognition and the speaker recognition are combined.
The method according to claim 1,
A speech detection process for detecting speech in the entire audio using the average energy method and the zero crossing rate; And
Extracting voice feature data from the extracted voice;
Wherein the face recognition and the speaker recognition are combined.
14. The method of claim 13,
Detecting a voiced sound region in the entire audio through the average energy method;
Determining whether there is a fricative at the front end of the voiced region using the zero crossing rate; And
Determining a voice as a voice when a fricative exists in a front end of the voiced region;
Wherein the face recognition and the speaker recognition are combined.
[14] The method of claim 13,
Performing fast Fourier transform (FFT) on the extracted speech;
Filtering the fast Fourier transformed speech data as a Mel filter which is a uniform filter;
Discrete cosine transform (DCT) for eliminating the correlation of the filtered speech data;
Extracting voice feature data by applying MFCC (Mixed Content Signal Classification) to the DCT voice data; And
Normalizing the number of speech feature data for each frame applied to the MFCC;
Wherein the face recognition and the speaker recognition are combined.
The method according to claim 1,
A face recognition and a speaker recognition that combines a registered feature vector registered in advance and a face feature vector extracted from the face image as face similarity.
The method according to claim 1,
A face recognition and a speaker recognition that calculates a correlation value between pre-registered registered feature data and voice specific data extracted from the voice as voice similarity.
The method according to claim 1,
Outputting a normalized facial similarity normalized to a number within a range of 0 to 1, and outputting a normalized phoneme similarity normalized to a number within a range of 0 to 1; And
Applying an artificial neural network model for applying authentication to the artificial neural network model by applying the normalized face similarity and the normalized voice similarity;
Wherein the face recognition and the speaker recognition are combined.
[19] The method of claim 18,
A process of learning an artificial neural network model composed of an input layer, a hidden layer, and an output layer by applying experimental data and a learning algorithm that inputs desired output values into an artificial neural network together; And
Performing authentication by applying the normalized face similarity and the normalized voice similarity to the artificial neural network model learned by the learning algorithm;
Wherein the face recognition and the speaker recognition are combined.

Images