TWI832552B - Speaker identification system based on meta-learning applied to real-time short sentences in an open set environment - Google Patents

Abstract

The present invention trains a deep learning speaker model (ResNest speaker model) based on a meta learning (Meta learning) method, and allows the deep learning speaker model to undergo training in plural training episodes (episode) and inverse transfer modification of two loss functions. Each training scenario has a support set of long statements and a query set of short statements. The present invention can convert input speech into speaker embedding vectors through a single deep learning speaker model, so that our speaker identification system can identify different registered speakers only through the comparison of speaker embedding vectors. And effectively blocks the entry of deception attacks and impostors. Accordingly, the present invention has the characteristics of lightweight, instant response, applicable short statements and open sets, and can use low-cost embedded hardware to implement the system.

Description

Speaker identification system based on meta-learning applied to real-time short sentences in an open set environment

The present invention relates to voice technology, and in particular, to a speaker identification system that identifies registered speakers and blocks deception attacks and impostors.

The number of voice application products is increasing day by day. Therefore, speaker identification technology is very important for voice application products. It needs to be able to authenticate the identity of the speaker and prevent impersonation and theft, so as to meet the requirements of anti-theft and confidentiality.

In speaker identification, registered speakers are called target speakers, unregistered speakers are called imposters, and attacking with artificially fake voices (trying to pass authentication) is called spoofing Attacks (spoofing attacks) include recording playback (replay) and speech synthesis (speech synthesis). Recording playback refers to the playback of pre-recorded sentences of registered speakers, while speech synthesis refers to fake speech generated by artificial technology.

Speaker identification technology in an open set environment must be able to correctly identify the target speaker and prevent deception attacks and intrusions by impostors. For example, Taiwan Announcement No. TWI749709B "A Speaker Identification Method" includes three stages of identification. The first stage of identification is to detect whether a text-related test statement is a spoofing attack. The second stage of identification is to detect impostor intrusion on a text-independent test statement. The third stage of identification uses the speaker identification system to determine which registered speaker the irrelevant test sentence belongs to. If it is not a registered speaker, it is determined to be an impostor. The first two stages use different features and have their own binary classifiers. The third stage uses complex classifiers, through ensemble learning and unanimity rule, and with conditional retry. mechanism to determine whether the test sentence irrelevant to this article is a target speaker or an impostor.

However, this conventional method uses a classifier identification method, which requires a long training sentence to build a model. Once the registered speakers change, the system needs to be retrained. Moreover, multiple classifiers are used, so the model is difficult to make small. It cannot respond immediately, requires large storage space, and has high computational complexity, making it difficult to use low-cost embedded hardware to implement the system.

The main purpose of the present invention is to disclose a speaker recognition system that has the characteristics of lightweight, instant response, applicable short sentences and open set environment.

In order to achieve the above purpose, the present invention is a speaker identification system based on meta-learning applied to real-time short sentences in an open set environment. The speaker identification system includes a speaker embedding generator (speaker embedding generator), which is composed of a Mel-filter bank (MFB) that captures acoustic features and a deep learning It is composed of the ResNest speaker model. The deep learning speaker model converts the acoustic feature vector output by the Mel filter into a speaker embedding vector. The deep learning speaker model is trained using plural training episodes (episode) based on meta learning. Each training episode has a support set of long sentences and a query set of short sentences. . At the same time, an objective function composed of two different loss functions is designed so that the deep learning speaker model can learn in the global and local embedding spaces at the same time, and the gradient of the objective function is back-transmitted to modify the deep learning speaker model. .

The speaker recognition system converts the input speech of at least one registered speaker into at least one prototype vector corresponding to the at least one registered speaker through the speaker embedding vector generator to complete the registration. After completing the registration, the speaker recognition system can be used by multiple testers to log in for testing, and the following steps will be performed for each tester:

A spoofing attack identification step is to determine whether the tester's input voice is a spoofing attack. Different from the conventional method, the input voice (test sentence) does not need to be limited to be relevant to this article. The speaker recognition system converts the input speech of the tester into a tester embedding vector through the speaker embedding vector generator, and then calculates the tester embedding vector of the tester and the at least one registered speaker in the system. Cosine similarity between prototype vectors. If all cosine similarity values exceed a threshold, it is determined that the input voice of the tester is not a spoofing attack, otherwise the tester will be refused to log in; and

An impostor and registered speaker identification step is to determine whether the input voice of the tester is an impostor or any of the registered speakers. First, the speaker identification system randomly cuts the tester's input speech into three consecutive sound segments, and uses the speaker embedding vector generator to generate speaker embedding vectors for each of the three segments of sound. Then, the similarity between the speaker's embedding vector of each of the three sound segments and the prototype vector of each registered speaker in the system is calculated, and a score is generated respectively. When the highest scores obtained by the speaker embedding vectors of the three segments all correspond to the same registered speaker and are greater than a set threshold, the tester is judged to be this specific registered speaker, otherwise it is judged to be an impersonation. , and refuse the tester's login.

Accordingly, the present invention can identify whether the input speech of the tester is a spoofing attack by comparing the at least one registered speaker, and can directly identify which of the registered speakers it is, thereby effectively preventing spoofing. To attack and deny the login of impostors, the invention has the characteristics of lightweight, instant response, applicable short statements and open sets, and can use low-cost embedded hardware to implement the system.

The detailed description and technical content of the present invention are as follows with illustrations:

Please refer to "Figure 1" and "Figure 2". The present invention is a speaker identification system based on meta-learning applied to real-time short sentences in an open set environment. It includes a speaker embedding vector generator 10. The embedding vector generator 10 has a deep learning speaker model 101 and a Mel filter 102. The deep learning speaker model 101 converts the acoustic feature vector captured from the input speech 11 through the Mel filter 102. Embedding vector 12 for the speaker. The deep learning speaker model 101 has an objective function, and the characteristics of the objective function are: when different input speech sounds 11 produced by the same speaker are converted into speaker embedding vectors 12 after operation by the deep learning speaker model 101, different speech The speaker embedding vectors 12 have the characteristic of being gathered together to form a cluster in a high-dimensional space 20, and the clusters of different speakers are far apart from each other.

The deep learning speaker model 101 is trained using complex training episodes (episode) based on meta learning. At the same time, the objective function including two loss functions is designed, and the gradient of the objective function is back-transmitted to modify the deep learning. The speaker model 101 is used to reduce the value of the objective function. The two loss functions can choose Angular prototypical loss and Softmax loss. In the plural training plots, each training plot has a support set of long sentences and a query set of short sentences. When the support set is processed by the deep learning speaker model 101, a prototype will be generated. (prototype)vector21. Similarly, after the query set is processed by the deep learning speaker model 101, a speaker embedding vector 22 will be generated. The prototype vector 21 and the speaker embedding vector 22 will be located in the high-dimensional space 20. During the training of each training episode, the deep learning speaker model 101 can use the gradient of the inverse transfer objective function to not only modify the model in the local embedding space, but also obtain more information from the global embedding space. Conducive to learning.

Please refer to "Figure 3" again. The speaker recognition system converts the input speech 31 of at least one registered speaker 30 into at least one prototype vector corresponding to the at least one registered speaker 30 through the speaker embedding vector generator 10. 32 and the registration is completed, the prototype vector 32 can be regarded as a certain prototype vector 21 in the high-dimensional space 20 (as shown in Figure 2).

In order to avoid errors caused by a single operation, in practice, the speaker identification system can cut the input speech 31 of the registered speaker 30 into a plurality of words, and then convert them into a plurality of speaker embedding vectors respectively, and then calculate the average The value serves as the prototype vector 32 of the corresponding registered speaker 30. And when the length of the input voice 31 of the registered speaker 30 is insufficient, the speaker recognition system can copy the input voice 31 of the registered speaker 30 and increase the length of the voice to meet the requirement of cutting the input voice 31 of the registered speaker 30 Required length requirements.

After completing the registration, the speaker identification system allows multiple testers 40 to perform login testing, and performs a deception attack identification step and an impostor and registered speaker identification step for each tester 40 .

Please refer to "Figure 4" again. In the spoofing attack identification step, in order to determine whether the input voice 41 of the tester 40 is a spoofing attack 46, the speaker identification system first converts the input voice 41 of the tester 40 into Acoustic feature vector, and then generate a tester embedding vector 42, and then use a calculation step 43 to calculate whether the cosine similarity between the tester embedding vector 42 and the prototype vectors 32 of all registered speakers 30 exceeds a threshold, and then use A judgment step 44 is performed. If all exceed the threshold, it is judged that the input speech 41 of the tester 40 is a real person's pronunciation 45. If any cosine similarity value does not exceed the threshold, it is judged to be a spoofing attack 46. And denied the tester 40 logins.

In addition, the cosine similarity between the tester embedding vector 42 and the prototype vectors 32 of all registered speakers 30 can be expressed by the following equation:

in is the embedding vector 42 for this tester, and is the prototype vector 32 of all registered speakers 30 . The so-called spoofing attack 46 includes recording playback (replay) and speech synthesis (speech synthesis) to reproduce speech by electronic means. It is not a real person's pronunciation 45 . Therefore, the input speech 41 of the spoofing attack 46 is converted into the tester's embedding vector. 42, the cosine similarity with the prototype vector 32 of at least one registered speaker will not be high. Therefore, as long as the cosine similarity between the tester's embedding vector 42 and the prototype vector 32 of each registered speaker 30 is determined to be greater than this valve If the value is 45, it can be judged as a real person's pronunciation, otherwise it is a spoofing attack 46.

Please refer to "Figure 5" again. In the step of identifying the impostor and the registered speaker, it is determined that the input voice 41 of the tester 40 is an impostor 47 or any of the registered speakers 30. In practice, the input voice 41 is the input voice 31 in the previous step. In this step, we randomly cut the input voice 41 of the tester 40 into three consecutive sound segments 411. The speaker embedding vector generator 10 converts the three segments of the voice 411 of the tester 40 into acoustic feature vectors to generate three segments of speaker embedding vectors 412, and then uses a calculation step 43A to calculate the speech of each segment respectively. The similarity between the speaker embedding vector 412 and the prototype vector 32 of the at least one registered speaker 30 is determined, and a score is generated respectively.

In addition, the similarity measurement method here is different from the cosine similarity mentioned above. To be more clear, in the step of identifying the impostor and the registered speaker, the similarity measurement method is based on the input speech vector length. Cosine similarity scaled by the length of the input speech vector. The similarity between the speaker embedding vector 412 of the i-th segment and the prototype vector 32 of the at least one registered speaker 30 can be expressed by the following equation:

in is the speaker embedding vector 412, and is the prototype vector 32 of all registered speakers 30 . Similarity The corresponding score is obtained by using the Softmax operation to convert the similarity into a probability distribution, which can be expressed by the following equation:

Through the above calculations, the similarity value obtained by the measurement method can effectively increase the difference in score distribution between speaker embedding vectors and prototype vectors of different registered speakers, thereby increasing discrimination.

Then a judgment step 44A is used to identify the impostor and the registered speaker. When the highest scores obtained by the speaker embedding vectors 412 of the three segments all correspond to the same registered speaker and are greater than a set threshold, it is called In order to satisfy all the requirements, the tester 40 can be judged to be the specific registered speaker 30, otherwise it will be judged to be an impostor 47, and the tester 40 will be refused to log in.

In the step of identifying the impostor and the registered speaker, in the speaker embedding vector 412 of the three segments, if the scores of the speaker embedding vector 412 of only two segments and the prototype vector 32 of the registered speaker are greater than the setting The threshold cannot satisfy the full decision, and if the scores of the speaker's embedding vector 412 of the remaining fragment and the prototype vectors 32 of other registered speakers are both less than a specified threshold, we call this situation a majority decision, and the registered speaker 30 corresponding to the speaker embedding vectors 412 of the two segments is called the registered speaker 30 of the majority decision. In the case of a majority, it conveys a low probability that the tester 40 is not the registered speaker of the majority 30. In practice, the values of the set threshold and the specified threshold can be adjusted according to usage requirements and test results to optimize the identification results.

In order to increase the recognition rate of the registered speaker 30, when the impostor and the registered speaker do not meet the full majority rule in the identification step, but still meet the majority rule, then in the deception attack identification step, the registered speaker with the highest cosine similarity will If the speaker 30 is the same as the registered speaker 30 corresponding to two of the speaker embedding vectors in the step of identifying the impostor and the registered speaker (that is, the registered speaker 30 in the majority vote), for the sake of caution , retry 48 can be performed, that is, an opportunity is given to re-perform the step of identifying the impostor and the registered speaker.

In order to increase the blocking rate of the impostor 47, in the spoofing attack identification step, the registered speaker 30 corresponding to the highest cosine similarity is the same as the registered speaker 30 judged in the impostor and registered speaker identification step. 30. If they are not the same, the tester 40 will be refused to log in.

Please refer to "Figure 6" again. Next, we use UMAP projection to visualize the distribution of speaker embedding vectors 50A, 50B, 50C, and 50D. Four speakers were randomly selected for comparison, each speaker had 60 utterances, each utterance was 3 seconds in length. As shown in the figure below, the left figure is the traditional speaker embedding vector (i-vector) distribution. The speaker embedding vectors 50A, 50B, 50C, and 50D of the same speaker are scattered, and the speaker embeddings of different speakers are scattered. Vectors 50A, 50B, 50C, and 50D are also not sufficiently separated. In contrast, the image on the right is the distribution of speaker embedding vectors generated by our deep learning speaker model. It can be observed that the speaker embedding vectors 50A, 50B, 50C, and 50D of the same speaker are concentrated in one place, and the speaker embedding vectors 50A, 50B, 50C, and 50D of different speakers are far enough apart. That is to say, in the speaker embedding space learned by our model, the speech of different speakers can be clearly distinguished.

In order to more clearly demonstrate the difference in efficacy between the present invention and the conventional technology, comparisons are made with respect to the pass rate of registered speakers, the blocking rate of impostors, and the detection rate of spoofing attacks.

In the experiment on the pass rate of registered speakers, we recruited 5 registered speakers. Each registered speaker records 20 minutes of speech into the system microphone. In addition, in order to evaluate the recognition rate of different test sentence lengths, we further divided the 20 minutes of speech of each registered speaker into three lengths of 1, 2 and 3 seconds, each with 400 sentences.

In the experiment of impostor blocking rate, we recruited 20 unregistered whisperers as impostors. Each speaker records a 5-minute test sound file into the system's microphone. To evaluate the performance comparison between different test speech lengths, we further divided each impostor's 5 minutes of speech into three lengths of 1, 2, and 3 seconds, with 100 utterances in each length.

In the spoofing attack blocking rate experiment, we recorded 5 minutes of speech for each registered speaker as a test corpus for spoofing attacks. We split the recorded corpus into three lengths: 1 second, 2 seconds and 3 seconds. First, each speaker's 5-minute corpus is divided into 100 audio files, each of which is 3 seconds long. Then, for each 3-second audio file, we randomly select continuous audio of 2-second and 1-second lengths, so the number of 1-second and 2-second voices is also 100.

The conventional technology adopts the technology disclosed in Taiwan Announcement No. TWI749709B "A Speaker Identification Method", which uses an i-vector speaker identification model. The experimental results list is as follows:

It can be seen that this case can still achieve quite good results compared with conventional wisdom in short sentences (1 second). 1 second 2 seconds 3 seconds Pass rate of registered speakers of the present invention 96.85% 98.13% 98.46% Registered speakers of Xi Zhi pass 75.56% 77.94% 80.88% 1 second 2 seconds 3 seconds Impostor blocking rate of the present invention 99.92% 99.85% 99.76% Impostor blocking rate 80.13% 86.95% 91.62% 1 second 2 seconds 3 seconds Deception attack detection rate of the present invention 99.87% 99.66% 99.31% Common knowledge about spoofing attack detection rate 94.86% 95.91% 98.44%

To sum up, the present invention has the following characteristics:

1. Compared with the traditional identification method using classifiers, the present invention allows registered speakers to complete system registration with a small number of enrolled utterances, and thereby obtain reliable performance on short test sentences. Furthermore, our model does not require retraining even when any existing registered speakers are removed or new registered speakers are added to the system. In other words, any change in the target speaker set does not require retraining the model.

2. A single lightweight speaker model can be applied in an open set environment and handle registered speaker identification, spoofing attack detection and impostor blocking at the same time. This lightweight feature is not only conducive to low-cost embedded hardware implementation, but also provides real-time response due to its lower computational complexity.

10: Speaker embedding vector generator 101: Deep Learning Speaker Model 102: Mel filter 11: Input voice 12: Speaker embedding vector 20:High-dimensional space 21:Prototype vector 22: Speaker embedding vector 30: Registered speaker 31: Input voice 32: Prototype vector 40:Tester 41: Input voice 411: Fragment of sound 412: Speaker embedding vector 42:Tester embedding vector 43, 43A: Calculation steps 44, 44A: Judgment steps 45: Real person pronunciation 46: Spoofing attack 47:Impostor 48:Retry 50A, 50B, 50C, 50D: speaker embedding vector

Figure 1 is a schematic diagram of the operation of the speaker embedding vector generator of the present invention. Figure 2 is a schematic diagram of speaker embedding vectors and prototype vectors in a high-dimensional embedding space according to the present invention. Figure 3 is a schematic diagram of speaker registration according to the present invention. Figure 4 is a schematic diagram of the deception attack identification steps of the present invention. Figure 5 is a schematic diagram of the steps for identifying impostors and registered speakers according to the present invention. Figure 6 is a comparison diagram of speaker embedding vector distributions of the traditional and the present invention.

10: Speaker embedding vector generator

101: Deep Learning Speaker Model

102: Mel filter

11: Input voice

12: Speaker embedding vector

Claims (7)

A speaker identification system based on meta-learning applied to real-time short sentences in an open set environment, which includes a speaker embedding vector generator with a deep learning speaker model and a Mel filter, The deep learning speaker model converts the acoustic feature vector output by the Mel filter into a speaker embedding vector. The deep learning speaker model is trained using complex training episodes based on a meta-learning method. Each training episode has a long The support set of sentences and the query set of short sentences are designed at the same time. An objective function composed of two different loss functions is used to modify the deep learning speaker model by back-propagating the gradient of the objective function. The speaker identification system uses The speaker embedding vector generator converts the input speech of at least one registered speaker into at least one prototype vector corresponding to the at least one registered speaker, and completes the registration. After the registration is completed, the speaker identification system can be used by multiple testers. test, and perform the following steps for each such tester: A spoofing attack identification step. In order to determine whether the tester's input speech is a spoofing attack, the speaker identification system converts the tester's input speech into a tester embedding vector through the speaker embedding vector generator, and then calculates the Whether the cosine similarity between the tester's embedding vector and the prototype vector of the at least one registered speaker exceeds a threshold. If both exceed the threshold, it is determined that the input speech of the tester is not a spoofing attack. Otherwise, Deny the tester login; and An impostor and registered speaker identification step. In order to determine whether the input voice of the tester is an impostor or any of the registered speakers, the speaker identification system randomly cuts the input voice of the tester into three consecutive sound segments. , and generate corresponding three-segment speaker embedding vectors through the speaker embedding vector generator, and then calculate the similarity between the speaker embedding vector of each segment and the prototype vector of each registered speaker, and A score is generated respectively. When the highest score value obtained by the speaker embedding vector of the three segments all corresponds to the same registered speaker and is greater than a set threshold, the tester is judged to be this specific registered speaker. Otherwise, it will be judged as an impostor and the tester will be refused to log in. The speaker identification system as described in claim 1, wherein the speaker identification system cuts the input speech of the registered speaker into a plurality of words, converts them into speaker embedding vectors respectively, and then calculates the average value as the corresponding The prototype vector of this registered speaker. The speaker identification system of claim 2, wherein the speaker identification system copies the input speech of the registered speaker and increases the length of the speech to meet the length requirement required for cutting the input speech of the registered speaker. The speaker identification system as described in claim 1, wherein in the step of identifying the impostor and the registered speaker, the speaker embedding vectors of the three segments are included in the speaker embedding vectors. If the speaker embedding vectors of two of the segments are the same as the registered speaker If the score of the prototype vector of the speaker is greater than the set threshold, and if the scores of the speaker's embedding vector of the remaining segment and the prototype vectors of other registered speakers are both less than a specified threshold, an opportunity will be given to retry the process. In the step of identifying impostors and registered speakers, the specified threshold is not greater than the set threshold. The speaker identification system as claimed in claim 1, wherein in the step of identifying the impostor and the registered speaker, in the speaker embedding vectors of the three segments, if only two of the speaker embedding vectors match the specific registered speaker If the score of the speaker's prototype vector is greater than the set threshold, then in the spoofing attack identification step, the registered speaker with the highest cosine similarity will be the same as the impostor and the registered speaker in the identification step. If the registered speaker corresponding to the speaker embedding vector is the same, an opportunity is given to redo the identification step between the impostor and the registered speaker. The speaker identification system of claim 1, wherein in the spoofing attack identification step, the registered speaker corresponding to the highest cosine similarity is the same as the impostor and the registered speaker determined in the identification step. If the registered speakers are different, the tester will be refused to log in. The speaker identification system of claim 1, wherein in the step of identifying the impostor and the registered speaker, the similarity measurement method is based on cosine similarity with a scale of input speech vector length, and the score is calculated using The softmax operation converts the similarity into a probability distribution.

Images