KR102094488B1 - Apparatus and method for user authentication using facial emg by measuring changes of facial expression of hmd user - Google Patents

Abstract

A user's personal authentication device and method using the measured EMG when a facial expression change of a HMD wearer is presented. The personal authentication device of a user according to an embodiment includes: an input unit that acquires an EMG signal of the face according to a change in the facial expression of the user; And a classifying unit for classifying facial expressions and users using a classifier previously learned according to facial expressions from the obtained EMG signal of the facial unit, and performing personal authentication of the user by recognizing a user classified through the classifier.

Description

A user's personal authentication device and method using the measured EMG when the facial expression of an HMD wearer changes {APPARATUS AND METHOD FOR USER AUTHENTICATION USING FACIAL EMG BY MEASURING CHANGES OF FACIAL EXPRESSION OF HMD USER}

The following embodiments relate to a user authentication technology using EMG on the face, and more particularly, to a personal authentication device and method of a user using EMG measured when a facial expression change of a HMD (Head-Mounted Display) wearer is changed.

The user authentication technology is a technology that verifies the user's identity using a password set by the user himself or the user's unique biosignal characteristics. Due to the spread of the Internet and the development of devices such as smart phones and tablet PCs, it has been possible to easily access innumerable information from anywhere. This is required.

For example, the password-based user authentication technology checks each other's knowledge set in advance by the user and the server, and it is easier to use and less expensive to build than other authentication methods, but it is easy to remember and use the same knowledge for multiple authentication systems. The disadvantage is that security is weak due to the characteristics that it is written similar to one's own information.

On the other hand, the biometric-based authentication method is a method of authenticating using the user's unique biometric information, which is convenient because it does not require a medium to remember or carry a security card like a password method. Currently, fingerprint recognition, face recognition, iris recognition, and vein recognition are used as the user authentication technology based on bio-signals.

On the other hand, EMG is an electrical signal generated according to muscle movement, and EMG-based interfaces have been developed for wheelchair control of the disabled, robot prosthesis control, paralyzed upper and lower limb rehabilitation robots. In addition, facial recognition technology using facial electromyography has recently been developed. However, the facial recognition technology mainly using the camera cannot be used because the user's face is obscured by the VR / AR device in a VR (Virtual Reality) / AR (Augmented Reality) environment using HMD.

 Cannan, J., & Hu, H. (2013, July). Automatic user identification by using forearm biometrics. In Advanced Intelligent Mechatronics (AIM), 2013 IEEE / ASME International Conference on (pp. 710-715). IEEE.  Yamaba, H., Kurogi, T., Aburada, K., Kubota, S. I., Katayama, T., Park, M., & Okazaki, N. (2017). On applying support vector machines to a user authentication method using surface electromyogram signals. Artificial Life and Robotics, 1-7.

The embodiments describe the user's personal authentication device and method using the EMG measured when the facial expression of the HMD wearer changes, and more specifically, perform the user's personal authentication using the measured EMG measured by the HMD wearer when the facial expression changes. Provides the technology to

Embodiments can quickly and accurately authenticate the user by acquiring the EMG signal of the facial part according to the user's facial expression change and classifying the expression and the user, and in the case of facial recognition technology mainly using a camera, a user in a VR / AR environment using HMD The face of the screen cannot be used because it is hidden by the VR / AR device, but the user's personal authentication can be performed using the EMG measured on the face when the facial expression of the HMD wearer changes through sensing.

The personal authentication device of a user according to an embodiment includes: an input unit that acquires an EMG signal of the face according to a change in the facial expression of the user; And a classifying unit for classifying facial expressions and users using a classifier previously learned according to facial expressions from the obtained EMG signal of the facial unit, and performing personal authentication of the user by recognizing a user classified through the classifier.

A feature extraction unit for extracting features from the acquired EMG signal of the face portion; And a user recognition unit for recognizing a user by classifying facial expressions using a classifier pre-trained according to an expression in the EMG signal obtained from the classification unit, wherein the classification unit extracts features from the feature extraction unit. According to the facial expression, the facial expression may be classified using a classifier previously learned.

The feature extracting unit, after performing pre-processing using a bandpass filter, extracts a feature by moving a size of a predetermined window, and the classification unit expresses an expression using Linear Discriminant Analysis (LDA), one of machine learning algorithms. And classifying the user, and the user recognition unit recognizes the most classified result up to the current point in time by using a majority vote, thereby using the facial EMG measured when the facial expression of the HMD wearer changes to authenticate the user. You can do

According to another embodiment of the present invention, a user's personal authentication method includes acquiring an EMG signal on the face according to a change in the facial expression of the user; And classifying the facial expression and the user using a classifier previously learned according to the facial expression from the obtained EMG signal of the face, and performing personal authentication of the user by recognizing the classified user through the classifier.

Extracting a feature from the acquired EMG signal of the face; And recognizing a user by classifying facial expressions using a classifier previously learned according to facial expressions from the EMG signal obtained from the classification unit, and classifying the classifiers previously learned according to facial expressions from the EMG signal of the facial unit. The step of classifying the facial expressions and the user may classify the facial expressions using the classifiers previously learned according to the facial expressions of the features extracted by the feature extraction unit.

According to embodiments, a personal authentication device and method for a user using a measured EMG when a facial expression change of a HMD wearer is performed using a facial EMG measured when a facial expression change of the HMD wearer is changed can be provided. .

According to embodiments, a user can quickly and accurately authenticate a user by acquiring an EMG signal on the face according to a user's facial expression change and classifying the expression and the user, and in the case of facial recognition technology mainly using a camera, VR / AR environment using HMD The user's face cannot be used because it is hidden by the VR / AR device, but the user's personal authentication can be performed using the EMG measured on the face when the facial expression of the HMD wearer changes through sensing.

1 is a view for explaining the structure of the user's personal authentication device using the measured EMG when the facial expression change of the HMD wearer according to an embodiment.
2 is a view showing an example of a facial expression photograph according to an embodiment.
3 is a diagram illustrating a user's personal authentication test method using EMG measured when a facial expression of a HMD wearer is changed according to an embodiment.
FIG. 4 is a diagram for explaining a user's personal authentication method using EMG measured when a facial expression of an HMD wearer changes according to an embodiment.
5 is a view for explaining a method of performing cross verification according to an embodiment.
6 is a view showing an example of a sensor layout according to an embodiment.

Hereinafter, embodiments will be described with reference to the accompanying drawings. However, the described embodiments may be modified in various other forms, and the scope of the present invention is not limited by the embodiments described below. In addition, various embodiments are provided to more fully describe the present invention to those skilled in the art. The shape and size of elements in the drawings may be exaggerated for a more clear description.

The present invention relates to a user authentication technology using an electromyography (Electromyograph: EMG). Recently, the EMG-based interface has been spotlighted by the development of Head-Mounted Display (HMD) such as VR / AR. In particular, since facial expressions based on EMG facial expressions cannot be used for camera-based facial expression recognition because the user's face is obscured, it is expected to be actively utilized in VR / AR applications.

Embodiments of the present invention can provide a user authentication technology utilized as a user's facial electromyography for seamless use of other muscle machine interfaces as well as a facial EMG-based facial expression recognition interface. Among the biometric technologies, since the EMG signal varies according to the frequency and waveform of each user and the degree of contraction and relaxation of the muscle, the EMG signal can be usefully used to authenticate or identify an individual if the EMG signal is measured with an automated device.

In particular, no user authentication studies have been conducted using EMG on the face, and a small user authentication study using EMG on the arm has been conducted. As shown in the table below, it can be confirmed that the gesture registration time for user authentication takes considerably and the accuracy of user authentication is not high.

Table 1 shows a comparison of EMG-based user authentication studies.

[Table 1]

As shown in Table 1, the existing studies (non-patent document 1 and non-patent document 2) are user authentication results using electromyography of other parts, which is difficult to compare with the electromyogram obtained from the face, but gesture registration for authentication as a whole. It can be seen that it takes a long time and the authentication accuracy is low.

The present invention provides a user authentication technology that can reduce the gesture registration time than previous studies and improve accuracy.

1 is a view for explaining the structure of the user's personal authentication device using the measured EMG when the facial expression change of the HMD wearer according to an embodiment.

Referring to FIG. 1, a personal authentication device 100 of a user using EMG measured when a facial expression of a HMD wearer changes according to an embodiment may include an input unit 110 and a classification unit 130, and is implemented. According to an example, the feature extraction unit 120 and the user recognition unit 140 may be further included.

The input unit 110 may obtain the EMG signal of the face according to a change in the facial expression of the user.

The feature extraction unit 120 may extract features from the acquired EMG signal of the face. Here, the feature extraction unit 120 may perform a pre-processing using a band pass filter, and then extract a feature by moving the size of a predetermined window.

The classification unit 130 may classify the facial expression and the user using a classifier previously learned according to the facial expression from the acquired EMG signal. In other words, the classification unit 130 may classify the facial expressions using the classifiers previously learned according to the facial expressions of the features extracted from the feature extraction unit 120. At this time, the classification unit 130 may classify facial expressions and users using Linear Discriminant Analysis (LDA), which is one of the machine learning algorithms.

In addition, the user recognition unit 140 is capable of performing personal authentication of the user by recognizing the user classified through the classifier, and uses the classifier previously learned according to the facial expression from the EMG signal obtained from the classification unit 130. By classifying facial expressions, the user can be recognized.

More specifically, the user recognition unit 140 uses the majority vote to recognize the most classified result up to the current time as the end user, so that the user's personal authentication is performed using the EMG measured at the face expression change of the HMD wearer's face expression. You can do

2 is a view showing an example of a facial expression photograph according to an embodiment.

As shown in FIG. 2, for the personal authentication experiment of the user using the EMG measured when the facial expression changes of the HMD wearer according to an embodiment, the facial expression picture used in the experiment, for example, delivers emotion well in the Radboud aces Database Photo stimuli that have been evaluated as being available can be used, and facial expressions such as biting, kissing, and frowning can be experimented with by directly photographing and using them.

3 is a diagram illustrating a user's personal authentication test method using EMG measured when a facial expression of a HMD wearer is changed according to an embodiment.

For example, to a total of 15 users 310, as shown in FIG. 3, electrodes can be attached to the HMD type array to measure the EMG of the facial area using Biosemi, a bio-signal measurement device.

First, the user 310 may be positioned toward the display device after attaching a plurality of sensors to the face. Thereafter, the user 310 may be guided by pressing the start button 321 through the display device. At this time, the pressing of the start button with a letter or the like may be guided 321 through the display unit of the display device or may be guided by voice through the speaker, or simultaneously through the display unit and the speaker.

When the user 310 presses the start button, one of 11 facial expressions is randomly selected through the display device, and a specific facial expression 322 is presented, and the user 310 as a subject follows the specific facial expression 322 presented for 3 seconds. Build Here, 11 facial expressions were used as examples, but the number of facial expressions may be changed.

In this case, the facial expression rest period may be guided 323 to the user 310 through the display device. For example, the facial expression rest may be guided 323 by letters or the like through the display unit of the display device, or may be guided by voice through the speaker, or simultaneously through the display unit and the speaker.

It is possible to create a database (DB) by repeating 10 times by making 11 facial expressions one time.

Hereinafter, the method of extracting the EMG feature of the face and evaluating the performance will be described in more detail.

EMG is generally analyzed in the time domain and frequency domain. In the present invention, the EMG analysis method in the time domain was used for real-time processing of the user authentication algorithm, and the following 12 existing general characteristics (approximate entropy (ApEn), cepstral coefficients (CC), difference absolute standard deviation value (DASDV) ), integral absolute value (IAV), mean absolute value (MAV), modified mean absolute value 1 (MAV1), modified mean absolute value 2 (MAV2), maximum fractal length (MFL), root mean square (RMS), sample entropy (SampEn), v-order (V), and waveform length (WL) were calculated. Since the performance of a classification algorithm based on multiple features is better than a classification algorithm based on a single feature, in the present invention, the extracted single features are composed of 2-4 different combinations of all the different features to optimize the combination of features optimized for the EMG signal. I found it.

(A) ApEn

The characteristic representing the approximate entropy of a signal is defined by the following equation.

[Equation 1]

(B) CC

It represents the astral coefficient of the signal and is defined by the following equation.

[Equation 2]

(C) DASDV

It represents the difference between the standard deviations between two adjacent samples as the absolute difference standard deviation, and is defined by the following equation.

[Equation 3]

(D) IAV

It is an integral of the absolute value of a signal for a certain period of time and is defined by

[Equation 4]

(E) MAV

It is a feature representing the average absolute value and is defined by the following equation.

[Equation 5]

(F) MAV1

It is a characteristic representing the modified absolute value average 1 and is defined by the following equation.

[Equation 6]

(G) MAV2

It is a characteristic representing the modified absolute value average 1 and is defined by the following equation.

[Equation 7]

(H) MFL

It is a feature representing the maximum fractal length and is defined by the following equation.

[Equation 8]

(I) RMS

It represents the square root of the square mean of the signal and is defined by the following equation.

[Equation 9]

(J) SampEn

A characteristic representing sample entropy is defined by the following equation.

[Equation 10]

(K) V

The V-order is a characteristic representing muscle contractility and is defined by the following equation.

[Equation 11]

(L) WL

It represents the length of the signal and is defined by the following equation.

[Equation 12]

FIG. 4 is a diagram for explaining a user's personal authentication method using EMG measured when a facial expression of an HMD wearer changes according to an embodiment.

Referring to FIG. 4, the user's personal authentication method according to an embodiment may include acquiring a facial EMG signal according to a change in a user's facial expression, and using a classifier previously learned according to an expression from the acquired EMG signal. It includes the step of classifying the facial expression and the user, and can recognize the user classified by the classifier to perform the user's personal authentication.

In addition, the user's personal authentication method according to an embodiment extracts a feature from the acquired EMG signal of the facial part, and uses the classifier previously learned according to the expression from the EMG signal of the facial part to obtain an expression. The classification may further include recognizing a user.

The personal authentication method of the user using the measured EMG when the facial expression changes of the HMD wearer according to an embodiment is the personal authentication device of the user using the measured EMG when the facial expression changes of the HMD wearer according to the embodiment described in FIG. 100). The personal authentication device 100 of the user using the EMG measured when the facial expression of the HMD wearer changes according to an embodiment may include an input unit 110 and a classification unit 130, and a feature extraction unit according to an embodiment 120 and the user recognition unit 140 may be further included.

First, the input unit 110 may obtain an EMG signal of the face according to a change in the facial expression of the user. For example, the input unit 110 may perform personal authentication of the user by using the EMG measured on the face when the facial expression of the HMD wearer changes.

Before extracting features from the EMG signal of the facial part through the feature extraction unit 120, after performing the pre-processing 410 using a band pass filter, the window is extracted while moving the size of a predetermined window (420), Features may be extracted (430). In addition, the feature extraction unit 120 may extract the feature from the acquired EMG signal of the face unit 420.

For example, after performing pre-processing using a bandpass filter (20-450 Hz) (410), the size of the window of 0.1 sec is shifted by 0.05 sec (420), and the 12 features described above are extracted (430). )can do.

These features can achieve high accuracy by using a combination of a plurality of the 12 features described above. In particular, when 4 features (ApEn, V, MAV2, CC) are used, 96.83% of the highest user recognition is achieved. Indicate accuracy. This was verified through experiments below.

The classification unit 130 may classify the facial expression and the user using the classifier previously learned according to the facial expression from the acquired EMG signal of the facial part 440. In other words, the classification unit 130 may classify the facial expressions using the classifiers previously learned according to the facial expressions of the features extracted from the feature extraction unit 120. At this time, the classification unit 130 may classify the facial expression and the user 440 using Linear Discriminant Analysis (LDA), which is one of the machine learning algorithms.

Thereafter, performance evaluation may be performed.

The user recognition unit 140 may perform personal authentication of the user by recognizing the user classified through the classifier, and uses the classifier previously learned according to the facial expression from the EMG signal obtained from the classification unit 130. Users can be recognized by classifying facial expressions. More specifically, the user recognition unit 140 uses a majority vote 450 to recognize the most classified result up to the current time as the end user, so that the user uses the EMG measured on the face when the facial expression change of the HMD wearer is changed. Can perform personal authentication.

5 is a view for explaining a method of performing cross verification according to an embodiment.

Referring to FIGS. 4 and 5, cross-validation is performed (460), and among 10 databases (DB) for each user, 4 training DBs and 6 test DBs are used to identify each expression. The subject recognition accuracy can be calculated.

In order to select the features optimized for the EMG signal, the recognition accuracy is calculated for the number of all cases that can be configured by combining 2 to 12 single features described above to optimize the user's personal authentication based on the EMG signal. Selected features can be selected.

As a result of the experiment, Tables 1 to 4 show the top 10 feature combinations showing the highest user recognition accuracy based on 'happiness'.

Table 1 shows the top 10 feature combinations (one feature) with high accuracy.

[Table 1]

Table 2 shows the top 10 feature combinations (2 features) with high accuracy.

[Table 2]

Table 3 shows the top 10 feature combinations (3 features) with high accuracy.

[Table 3]

Table 4 shows the top 10 feature combinations (4 features) with high accuracy.

[Table 4]

Referring to Table 1 to Table 4, when using four features (ApEn, V, MAV2, CC) it showed the highest accuracy of 96.83%.

Table 5 shows the results of user recognition accuracy according to facial expressions. More specifically, Table 5 shows user recognition accuracy for 11 facial expressions in 20 iterations when four features (ApEn, V, MAV2, and CC) are used. This was verified through experimental results as described above.

[Table 5]

For each expression, 15 users were recognized with an average accuracy of 71.00 to 96.83%, and the highest accuracy in 'happiness' was 96.83%.

Table 6 shows the results of user recognition accuracy for each expression according to the number of training DB changes. More specifically, Table 6 shows the average accuracy of user recognition of 10 iterations according to the number of training DBs used.

[Table 6]

Accordingly, it is possible to confirm the feature combination optimized for the EMG of the face.

According to embodiments of the present invention, a user authentication method using an electromyography of the face may be provided. Therefore, the method according to the present embodiment will be effectively used for the EMG interface of the facial part of the HDM environment.

6 is a view showing an example of a sensor layout according to an embodiment.

Referring to FIG. 6, a sensor may be attached to a colored portion. At this time, the potential difference between the two sensors attached to the same color can be used as each signal.

Twelve types of features are extracted from the user's facial expression EMG signal, and the classifier can be trained using the extracted features.

As described above, according to the embodiments, the HMD wearer's personal authentication device and method using the measured EMG when the facial expression change of the HMD wearer performs personal authentication of the user using the measured facial EMG when the facial expression changes of the HMD wearer Can provide. According to embodiments, the user can quickly and accurately authenticate the user by acquiring the EMG signal of the facial part according to the change in the facial expression of the user and classifying the facial expression and the user. In, the user's face cannot be used because it is hidden by the VR / AR device, but the user's personal authentication can be performed using the EMG measured in the face when the facial expression of the HMD wearer changes through sensing.

The device described above may be implemented with hardware components, software components, and / or combinations of hardware components and software components. For example, the devices and components described in the embodiments include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor (micro signal processor), a microcomputer, a field programmable array (FPA), It may be implemented using one or more general purpose computers or special purpose computers, such as a programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions. The processing device may run an operating system (OS) and one or more software applications running on the operating system. In addition, the processing device may access, store, manipulate, process, and generate data in response to the execution of the software. For convenience of understanding, a processing device may be described as one being used, but a person having ordinary skill in the art, the processing device may include a plurality of processing elements and / or a plurality of types of processing elements. It can be seen that may include. For example, a processing device may include a plurality of processors or a processor and a controller. In addition, other processing configurations, such as parallel processors, are possible.

The software may include a computer program, code, instruction, or a combination of one or more of these, and configure the processing device to operate as desired, or process independently or collectively You can command the device. Software and / or data may be interpreted by a processing device, or to provide instructions or data to a processing device, of any type of machine, component, physical device, virtual equipment, computer storage medium or device. Can be embodied in The software may be distributed on networked computer systems, and stored or executed in a distributed manner. Software and data may be stored in one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer readable medium. The computer-readable medium may include program instructions, data files, data structures, or the like alone or in combination. The program instructions recorded on the medium may be specially designed and constructed for the embodiments or may be known and usable by those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs, DVDs, and magnetic media such as floptical disks. -Hardware devices specially configured to store and execute program instructions such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include high-level language code that can be executed by a computer using an interpreter, etc., as well as machine language codes produced by a compiler.

As described above, although the embodiments have been described by a limited embodiment and drawings, those skilled in the art can make various modifications and variations from the above description. For example, the described techniques are performed in a different order than the described method, and / or the components of the described system, structure, device, circuit, etc. are combined or combined in a different form from the described method, or other components Alternatively, even if replaced or substituted by equivalents, appropriate results can be achieved.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (5)

An input unit for acquiring an EMG signal of the face according to a change in the facial expression of the user;
A feature extraction unit for extracting features from the acquired EMG signal of the face portion;
A classification unit for classifying facial expressions and users by using a classifier previously learned according to facial expressions from the acquired EMG signal of the facial unit; And
A user recognition unit that recognizes a user by classifying facial expressions using a classifier previously learned according to facial expressions from the EMG signal obtained from the classification unit
Including,
The feature extraction unit,
After performing pre-processing using a bandpass filter, characteristics are extracted by moving the size of a predetermined window, approximate entropy (ApEn), cepstral coefficients (CC), difference absolute standard deviation value (DASDV), integral absolute value ( IAV), mean absolute value (MAV), modified mean absolute value 1 (MAV1), modified mean absolute value 2 (MAV2), maximum fractal length (MFL), root mean square (RMS), sample entropy (SampEn), v- It is composed of a plurality of different combinations of single features of order (V) and waveform length (WL) to find a feature combination optimized for the EMG signal of the face,
The user's personal authentication device performs personal authentication of the user according to the recognition of the user classified through the classifier. According to claim 1,
The classification unit,
Classifying facial expressions using a classifier previously learned according to the facial expressions extracted from the feature extraction unit
Characterized in that, the user's personal authentication device. According to claim 1,
The classification unit,
We classify facial expressions and users using LDA (Linear Discriminant Analysis), one of the machine learning algorithms.
The user recognition unit,
Using the majority vote to recognize the most classified results up to the present time as the end user, performing personal authentication of the user by using the EMG measured in the face when the facial expression of the HMD wearer changes
Characterized in that, the user's personal authentication device. Acquiring an EMG signal on the face according to a change in the facial expression of the user;
Extracting a feature from the acquired EMG signal of the face;
Classifying an expression and a user using a classifier previously learned according to an expression from the acquired EMG signal of the facial part; And
Recognizing a user by classifying facial expressions using a pre-trained classifier according to facial expressions from the EMG signal obtained in the classification step
Including,
The step of extracting features from the EMG signal of the face portion,
After performing pre-processing using a bandpass filter, characteristics are extracted by moving the size of a predetermined window, approximate entropy (ApEn), cepstral coefficients (CC), difference absolute standard deviation value (DASDV), integral absolute value ( IAV), mean absolute value (MAV), modified mean absolute value 1 (MAV1), modified mean absolute value 2 (MAV2), maximum fractal length (MFL), root mean square (RMS), sample entropy (SampEn), v- It is composed of a plurality of different combinations of single features of order (V) and waveform length (WL) to find a feature combination optimized for the EMG signal of the face,
A user's personal authentication method, wherein the user's personal authentication is performed by recognizing a user classified through the classifier. According to claim 4,
In the facial EMG signal, the step of classifying the expression and the user using a classifier previously learned according to the expression,
The feature extracted in the step of extracting the feature is to classify the expression using a classifier previously learned according to the expression,
Recognizing the user,
Using the majority vote to recognize the most classified results up to the present time as the end user, performing personal authentication of the user by using the EMG measured in the face when the facial expression of the HMD wearer changes
Characterized in that, the user's personal authentication method.

Images