KR20230046946A - Electronic device for identifying a target speaker and an operating method thereof - Google Patents

Abstract

A method of operating an electronic device according to an embodiment includes the operations of: extracting voice characteristics of a target speaker based on a voice input; determining an utterance scenario of the voice input based on voice characteristics of the target speaker; obtaining final voice characteristics of the target speaker based on the determined utterance scenario; and determining whether the target speaker corresponds to a user based on the final voice characteristics of the target speaker and the final voice characteristics of the user. The utterance scenario may include a single speaker scenario and a multi-speaker scenario. In addition, various embodiments may be possible.

Description

Electronic device for identifying a target speaker and method for operating the same

Various embodiments of the present disclosure relate to an electronic device for identifying a target speaker and an operating method thereof.

Due to the spread of various electronic devices, the security of electronic devices is becoming more important. In particular, the electronic device should be unlocked only when a registered user uses the electronic device, preventing someone other than the user of the electronic device from using the electronic device.

For security of the electronic device, the user's voice may be used as the user's unique information. For example, voiceprint identification technology (or speaker verification technology) may be technology that uses a user's voice. The voiceprint recognition technology may extract voice features of a speaker from voice input and use the extracted voice features.

In general, voiceprint recognition technology is divided into two processes: voiceprint registration and voiceprint verification. During the registration process, the electronic device may register user information with the electronic device through the user's voice. During the verification process, the electronic device may extract voice features from the received voice and compare the extracted voice features with voice features of a pre-registered user. For example, the electronic device may calculate a similarity score between the extracted voice feature and the pre-registered voice feature of the user. The electronic device may determine that the speaker recognized from the voice and the registered user are the same person if the calculated similarity score is greater than the threshold value, and if the calculated similarity score is smaller than the threshold value, the speaker recognized from the voice and the registered user may be determined. can be judged as a different person.

Voiceprint recognition technology can perform voiceprint verification using two independent models (a voice extraction model and a voiceprint recognition model). Voiceprint recognition technology extracts information about a speaker's voice from an input voice through a voice extraction model, and then obtains a correspondence between a speaker and a user from information about a speaker's voice through a voiceprint recognition model. The second-stage voiceprint verification process uses two independent models, and it may not be easy to adjust the two models that perform the voiceprint verification process in a complex manner. Voiceprint recognition technologies each implementing independent models may have low recognition performance when receiving complex voices. A technique for accurately recognizing and verifying complex speech by complexly adjusting two models that perform different operations may be required.

Embodiments may provide a technology for accurately recognizing and verifying a complex voice through an end-to-end model, and for recognizing and verifying a target speaker even when an input voice does not include a designated keyword.

The technical problem to be achieved in this document is not limited to the technical problem mentioned above, and other technical problems not mentioned can be clearly understood by those skilled in the art from the description below. There will be.

An operating method of an electronic device according to an embodiment includes extracting a voice feature of a target speaker based on a voice input; determining an utterance scenario of the voice input based on the voice characteristics of the target speaker; obtaining a final speech feature of the target speaker based on the determined speech scenario; and determining whether the target speaker corresponds to the user based on final voice characteristics of the target speaker and final voice characteristics of the user, wherein the speech scenario may include a single speaker scenario and a multi-speaker scenario. there is.

According to an embodiment, the extracting may include obtaining an original voice feature based on the voice input; and extracting the voice features of the target speaker by inputting the original voice features and the intermediate embedded voice features of the user to a first network.

According to an embodiment, the first network receives the original speech feature and outputs a speaker extraction embedding feature for extracting the speech feature of the speaker included in the speech input. lution layer; a splicing layer that receives the speaker extraction embedding feature and the middle embedding voice feature of the user and outputs a splicing feature; a second convolution layer that receives the splicing feature and outputs a mask; and a multiplier receiving the mask and the speaker extraction embedding feature and outputting a speech feature of the target speaker.

According to an embodiment, the determining of the speech scenario of the voice input may include determining the speech scenario of the voice input by comparing the original voice characteristics and the voice characteristics of the target speaker.

According to an embodiment, the operation of determining the speech scenario of the voice input by comparing the original voice characteristics and the voice characteristics of the target speaker may include a mean square error between the original voice characteristics and the voice characteristics of the target speaker. error) is less than a threshold value, determining the speech scenario as a single speaker scenario, and determining the speech scenario as a multi-speaker scenario when the mean square error between the original speech feature and the target speaker's speech feature is greater than or equal to the threshold value. can include

According to an embodiment, the acquiring of the final voice characteristics of the target speaker may include, in response to the single speaker scenario, acquiring the final voice characteristics of the target speaker by inputting the original voice characteristics to a second network; Alternatively, in response to the multi-speaker scenario, an operation of inputting the voice characteristics of the target speaker to the second network and acquiring final voice characteristics of the target speaker may be included.

According to an embodiment, the second network may include a speaker embedding layer that receives the original speech feature or the speech feature of the target speaker and outputs an intermediate embedded speech feature of the target speaker and a speaker embedding layer of the target speaker. An attention statistics pooling layer may be included that receives intermediate speech features and outputs final speech features of the target speaker.

According to an embodiment, the checking whether the target speaker corresponds to the user may include calculating a similarity value between a final voice feature of the target speaker and a final voice feature of the user; and determining whether the target speaker corresponds to the user based on a calculation result.

According to an embodiment, the user's intermediate voice feature is obtained as a result of inputting the user's voice feature obtained based on the user's voice input to the speaker embedding layer, and the user's final voice feature may be obtained as a result of inputting the user's intermediate embedded voice features to the attention statistics pooling layer.

According to an embodiment, the first network and the second network may be jointly learned with a third network that transforms a speaker's voice feature based on an intermediate embedded voice feature of the speaker.

An electronic device according to an embodiment includes a memory including instructions; and a processor electrically connected to the memory and configured to execute the instructions, wherein when the instructions are executed by the processor, the processor extracts voice characteristics of a target speaker based on a voice input, Determine a speech scenario of the voice input based on the voice characteristics of the speaker, obtain final voice characteristics of the target speaker based on the determined speech scenario, and obtain final voice characteristics of the target speaker and final voice characteristics of the user It is checked whether the target speaker corresponds to the user, and the speaking scenario may include a single speaker scenario and a multi-speaker scenario.

According to an embodiment, the processor obtains an original speech feature based on the voice input, inputs the original voice feature and the intermediate embedded voice feature of the user to a first network, and extracts the voice feature of the target speaker. can do.

According to an embodiment, the first network may include: a first convolution layer that receives the original speech features and outputs speaker extraction embedding features for extracting speech features of a speaker included in the speech input; a splicing layer that receives the speaker extraction embedding feature and the user's middle embedding voice feature and outputs a splicing feature; a second convolution layer that receives the splicing feature and outputs a mask; and a multiplier receiving the mask and the speaker extraction embedding feature and outputting a speech feature of the target speaker.

According to an embodiment, the processor may determine a speech scenario of the voice input by comparing the original voice characteristics and the voice characteristics of the target speaker.

According to an embodiment, the processor determines that the speech scenario is a single speaker scenario when the mean square error between the original speech feature and the speech feature of the target speaker is smaller than a threshold value, and the original speech feature and the target speaker When the mean square error of the speech feature of is greater than or equal to a threshold value, the speech scenario may be determined as a multi-speaker scenario.

According to an embodiment, the processor obtains final voice features of the target speaker by inputting the original voice features to a second network in response to the single-speaker scenario, or in response to the multi-speaker scenario, the Final voice characteristics of the target speaker may be obtained by inputting the voice characteristics of the target speaker to the second network.

According to an embodiment, the second network may include: a speaker embedding layer that receives the original speech feature or the speech feature of the target speaker and outputs an intermediate embedded speech feature of the target speaker; and an attention statistics pooling layer that receives the middle embedded speech feature of the target speaker and outputs a final speech feature of the target speaker.

According to an embodiment, the processor may calculate a similarity value between a final voice feature of the target speaker and a final voice feature of the user, and determine whether the target speaker corresponds to the user based on the calculation result.

According to an embodiment, the user's intermediate voice feature is obtained as a result of inputting the user's voice feature obtained based on the user's voice input to the speaker embedding layer, and the user's final voice feature may be obtained as a result of inputting the user's intermediate embedded voice features to the attention statistics pooling layer.

According to an embodiment, the first network and the second network may be jointly learned with a third network that converts a speaker's voice feature based on an intermediate embedded voice feature of the speaker.

1 is a schematic block diagram of an electronic device for identifying a target speaker, according to an embodiment.
2 is an example of a speaker voice extraction model based on a neural network, according to an embodiment.
3 is an example of a speaker recognition model based on a neural network, according to an embodiment.
4 is a diagram for describing an operation of identifying a target speaker based on a voice input, according to an exemplary embodiment.
5 is an example for explaining learning of neural network-based models according to an embodiment.
6 is another example for explaining learning of neural network-based models according to an embodiment.
7 is a flow diagram of an operation to identify a target speaker, according to one embodiment.

Hereinafter, specific embodiments are provided for the reader's comprehensive understanding of the method, apparatus, and/or system described herein. After the disclosure of this application is understood, various changes, modifications, and equivalents of the methods, apparatus, and/or systems described herein will also become apparent. The order of operations described in this specification is only an example, and is not limited thereto, and the order may be changed except for operations to be performed in a specific order. Also, for clarity and conciseness, descriptions of features known in the art may be omitted.

Features described in the text may be implemented in other forms and should not be construed as being limited to the examples described in the text. To the contrary, the examples described herein are provided to illustrate only some of the many possible ways of implementing the methods, apparatus and/or systems described herein, many of which will become apparent after reading the present disclosure. .

As used herein, the term "and/or" includes any one and any combination of any two or more of the associated listed items.

Although terms such as “first,” “second,” and “third” may be used herein to describe various elements, elements, regions, layers, or sections, elements, elements, regions, layers, or sections It should not be limited by these terms (eg, terms such as "first", "second" and "third"). Terms such as “first,” “second,” and “third” are only used to distinguish one element, element, region, layer, or section from another element, element, region, layer, or section. Accordingly, without departing from the teachings of the examples, the first member, the first element, the first region, the first layer, or the first part mentioned in the examples described herein may be the second member, the second element, the second It may also be referred to as a region, a second layer, or a second portion.

In the specification, when an element (eg, layer, region, or substrate) is described as “existing” on, “connected to,” or “coupled to” another element, that element may “exist” directly on the other element or , may be directly "connected" or "coupled" to another element, or may have one or more other elements in between. Conversely, when an element is described as being “directly present” on, “directly connected to” or “directly coupled to” another element, there may be no other elements in between.

Terms used in the text are only used to describe various examples, and are not used to limit the disclosure. The singular forms are intended to include the plural forms unless the context clearly dictates otherwise. The terms “comprise,” “include,” and “have” describe the presence of the described feature, quantity, operation, component, element, and/or combination thereof, but describe the presence of one or more other features, quantities, operations, The presence or addition of components, elements, and/or combinations thereof is not excluded.

Unless otherwise defined, all terms (including technical terms and scientific terms) used in the text have the same meaning as commonly understood by a person of ordinary skill in the art to which the present invention belongs after understanding the present disclosure. Unless explicitly defined in the text, terms (eg, terms defined in a general dictionary) are to be interpreted as having a meaning consistent with their meaning in the context of the field and in this disclosure.

In addition, in describing the embodiments, if it is determined that a detailed description of a known related structure or function may obscure the description of the present invention, the detailed description will be omitted.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. In the description with reference to the accompanying drawings, the same reference numerals are given to the same components regardless of reference numerals, and overlapping descriptions thereof will be omitted.

1 is a schematic block diagram of an electronic device for identifying a target speaker, according to an embodiment.

According to an embodiment, the electronic device 100 may be a locked device for security. The electronic device 100 may unlock the electronic device 100 by comparing previously registered user voice information and newly acquired voice information. The electronic device 100 may extract information about the target speaker's voice based on the voice input, and determine whether the target speaker is the same as a pre-registered user based on the information about the target speaker's voice. A target speaker may be an object to be recognized and verified in a voice input. The electronic device 100 may provide a technology for accurately recognizing and verifying complex speech through an end-to-end model. The electronic device 100 may provide a technique for recognizing and verifying a target speaker even when an input voice does not include a designated keyword.

Referring to FIG. 1 , according to an embodiment, an electronic device 100 may include a processor 110 and a memory 130 . The processor 110 may process data stored in the memory 130 . The processor 110 may execute computer readable codes (eg, software) stored in the memory 130 and instructions triggered by the processor 110 . The processor 110 may be a data processing device implemented in hardware having a circuit having a physical structure for executing desired operations. For example, desired operations may include codes or instructions included in a program. For example, a data processing unit implemented in hardware includes a microprocessor, a central processing unit, a processor core, a multi-core processor, and a multiprocessor. , Application-Specific Integrated Circuit (ASIC), and Field Programmable Gate Array (FPGA).

According to an embodiment, the memory 130 may store data for calculation or calculation results. Memory 130 may store instructions (or programs) executable by processor 110 . For example, the instructions may include instructions for executing an operation of the processor 110 and/or an operation of each component of the processor 110 . The memory 130 may be implemented as a volatile memory device or a non-volatile memory device. The volatile memory device may be implemented as dynamic random access memory (DRAM), static random access memory (SRAM), thyristor RAM (T-RAM), zero capacitor RAM (Z-RAM), or twin transistor RAM (TTRAM). Non-volatile memory devices include electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic RAM (MRAM), spin-transfer torque (STT)-MRAM (conductive bridging RAM), and conductive bridging RAM (CBRAM). , FeRAM (Ferroelectric RAM), PRAM (Phase change RAM), Resistive RAM (RRAM), Nanotube RRAM (Polymer RAM (PoRAM)), Nano Floating Gate Memory Memory (NFGM)), holographic memory, molecular electronic memory device (Molecular Electronic Memory Device), or Insulator Resistance Change Memory. The memory 130 may store data.

According to an embodiment, the memory 130 may store information related to the user of the electronic device 100 . Information related to the user of the electronic device 100 may include the user's intermediate voice feature and the user's final voice feature. Hereinafter, a target speaker identification operation of the processor 110 will be described in detail.

According to an embodiment, the processor 110 may extract a voice feature of a target speaker based on a voice input. The processor 110 may obtain an original voice feature based on the voice input, and the processor 110 may obtain the original voice feature and the user's intermediate voice feature (eg, the user's intermediate embedded voice feature pre-stored) in the first network. It is possible to extract the speech characteristics of the target speaker by inputting .

According to an embodiment, the voice input may be a voice input for use of the electronic device 100 . Voice input may be generated in various speech scenarios. For example, voice input may be generated by a single speaker in a single speaker scenario. For another example, voice input may be generated by multiple speakers in a multi-speaker scenario. The voice input may include a single speaker's voice or multi-speaker's voice.

According to one embodiment, the target speaker may be a speaker to be recognized and verified in the voice input. The target speaker may correspond to a subject to be verified whether or not the same as the user of the electronic device 100 . For example, in a single speaker scenario, the target speaker may be the corresponding one person. For another example, in a multi-speaker scenario, a target speaker may be any one of a plurality of speakers. The target speaker may be a speaker who is subject to verification and has characteristics similar to those of registered users.

According to an embodiment, the middle embedding speech characteristics of the user may be pre-stored characteristics of the user. The user's intermediate embedding voice feature may be for extracting information about the target speaker from the voice input. The original voice feature and the voice feature of the target speaker may be voice features configured frame by frame (eg, frames generated over time). When the length of the voice (eg, the length on the time axis) is long, the size of data corresponding to the voice feature may also be large. For example, the original speech feature and the speech feature of the target speaker may be Mel-scale Frequency Cepstral Coefficients (MFCC). For another example, the original voice features and the target speaker's voice features may be FilterBank features. However, the above examples are only examples, and the present invention is not limited thereto. An operation of acquiring voice characteristics of the target speaker by the processor 110 will be described in detail with reference to FIG. 2 .

According to an embodiment, the processor 110 may determine a speech scenario of the voice input based on voice characteristics of the target speaker. Speech scenarios of voice input may include a single speaker scenario and/or a multi-speaker scenario. The processor 110 may determine an utterance scenario of the voice input by comparing the original voice characteristics and the voice characteristics of the target speaker. The processor 110 may determine an utterance scenario as a single speaker scenario when a mean square error between the original voice feature and the target speaker's voice feature is smaller than a threshold value. The processor 110 may determine the speech scenario as a multi-speaker scenario when the mean square error between the original speech feature and the target speaker's speech feature is greater than or equal to a threshold value. Since the processor 110 may determine a speech scenario of the input voice based on the difference between the voice characteristics of the target speaker and the original voice characteristics, the speech scenario of the voice input may be easily determined. The processor 110 may select an input suitable for the second network 300 by determining whether a speech input scenario is a single speaker scenario or a multi-speaker scenario. The processor 110 can guarantee voiceprint verification performance in a multi-speaker scenario, and can also guarantee voiceprint verification performance in a single speaker scenario. The processor 110 may provide a high-performance voiceprint verification service to the eye user.

According to an embodiment, the processor 110 may obtain final speech characteristics of the target speaker based on the determined speech scenario. In response to a single speaker scenario, the processor 110 may acquire final voice features of a target speaker by inputting original voice features to the second network. In response to a multi-speaker scenario, the processor 110 may acquire final voice characteristics of the target speaker by inputting voice characteristics of the target speaker to the second network. The final speech feature may have a fixed size regardless of the length of the speech. The final speech feature may be a one-dimensional vector. For example, the size of the final voice feature may be set to 1x128, 1x256, or 1x512. The operation of obtaining the final speech feature of the target speaker will be described in detail with reference to FIG. 3 .

According to an embodiment, the processor 110 may determine whether the target speaker corresponds to the user based on the final voice characteristics of the target speaker and the final voice characteristics of the user. The processor 110 may calculate a similarity value between the final voice feature of the target speaker and the final voice feature of the user, and determine whether the target speaker corresponds to the user based on the calculated similarity value. For example, the processor 110 may determine that the target speaker corresponds to the user when a similarity value (eg, a cosine similarity value) is greater than a threshold value. For another example, when the similarity value is less than or equal to the threshold value, it may be determined that the target speaker does not correspond to the user. Hereinafter, configurations and operations of the first network and the second network will be described in detail.

2 is an example of a speaker voice extraction model based on a neural network, according to an embodiment.

Referring to FIG. 2 , according to an embodiment, a processor (eg, the processor 110 of FIG. 1 ) may extract a voice feature of a target speaker from a voice input based on the first network 200 . The first network 200 may include a first convolution layer 210 , a splicing layer 220 , a second convolution layer 230 , and a multiplier 240 .

According to an embodiment, the first convolution layer 210 may receive original speech features and output speaker extraction embedding features for extracting speech features of a speaker included in the speech input. The splicing layer 220 may receive the speaker extraction embedding feature and the user's middle embedding voice feature, and output the splicing feature. The middle embedding voice feature of the user may be a pre-stored user feature, and may be for extracting information about a target speaker from a voice input. The second convolution layer 230 may receive a splicing feature and output a mask. The second convolution layer 230 may be a fully convolutional layer and may include a plurality of 1D-convolutional layers. Optionally, a normalization layer may be added between the splicing layer 220 and the second convolution layer 230 to normalize splice features. The multiplier 240 may receive the mask and speaker extraction embedding features and output voice features of the target speaker. The target speaker may be a speaker to be recognized and verified in the voice input, and the voice feature of the target speaker may be a voice feature configured frame by frame (eg, frames generated over time).

3 is an example of a speaker recognition model based on a neural network, according to an embodiment.

Referring to FIG. 3 , according to an embodiment, a processor (eg, the processor 110 of FIG. 1 ) determines the target speaker's final voice characteristics based on the original voice characteristics or the target speaker's voice characteristics based on the second network 300 . Voice characteristics can be obtained. The second network 300 may include a speaker embedding layer 310 and an attention statistics pooling layer 320.

According to an embodiment, the speaker embedding layer 310 may receive an original voice feature or a voice feature of a target speaker and output an intermediate embedded voice feature of the target speaker. The speaker embedding layer 310 may be constructed using a Sequeze and Excitation Block (SE) block, a ResNet block, or a Time Delay Neural Network (TDNN). However, the above examples are only examples, and the present invention is not limited thereto. The speaker embedding layer 310 may be another commonly used neural network or a combination thereof.

The attention statistics pooling layer 320 may receive the middle embedded speech feature of the target speaker and output the final speech feature of the target speaker. The final speech feature may have a fixed size regardless of the length of the speech. For example, the final speech feature may be a one-dimensional vector.

According to an embodiment, the processor 110 may select an input suitable for the second network 300 by determining whether a speech input scenario is a single speaker scenario or a multi-speaker scenario. The processor 110 can guarantee voiceprint verification performance in a multi-speaker scenario, and can also guarantee voiceprint verification performance in a single speaker scenario. The processor 110 may provide a high-performance voiceprint verification service to the eye user.

According to an embodiment, the processor 110 may obtain an intermediate voice feature of the user and a final voice feature of the user based on the second network 300 . The user's intermediate voice feature and the user's final voice feature may be acquired in the voiceprint registration step. The middle embedding voice feature of the user may be obtained as a result of inputting the user's voice feature to the speaker embedding layer 310 . The middle embedded speech feature of the user may be used when extracting the speech feature of the target speaker. The user's final voice feature may be obtained as a result of inputting the user's middle embedded voice feature to the attention statistics pooling layer 320 . The final voice feature of the user may be the one used when verifying whether the voice input contains the user's voice.

4 is a diagram for describing an operation of identifying a target speaker based on a voice input, according to an exemplary embodiment.

Referring to FIG. 4 , according to an embodiment, a processor (eg, the processor 110 of FIG. 1 ) may obtain original voice features from a voice input based on the feature extraction module 410 . The voice input may be a voice input for use of the electronic device 100 . The voice input may be generated in various speech scenarios, and may include a single speaker's voice or multiple speakers' voices.

According to an embodiment, the processor 110 may obtain voice features of the target speaker from the original voice features and the intermediate embedded voice features of the user based on the first network 200 . A target speaker may be a speaker to be recognized and verified in the voice input. The target speaker may correspond to a subject to be verified whether or not the same as the user of the electronic device 100 . The target speaker may be a speaker who is subject to verification and has characteristics similar to those of registered users. The original voice feature and the voice feature of the target speaker may be voice features configured frame by frame (eg, frames generated over time). When the length of the voice (eg, the length on the time axis) is long, the size of data corresponding to the voice feature may also be large. For example, the original voice feature and the voice feature of the target speaker may be Mel-scale Frequency Cepstral Coefficients (MFCC) or FilterBank features. However, the above examples are only examples, and the present invention is not limited thereto.

According to an embodiment, the first network 200 may include a first convolution layer 210 , a splicing layer 220 , a second convolution layer 230 , and a multiplier 240 . The first convolution layer 210 may receive original speech features and output speaker extraction embedding features for extracting speech features of a speaker included in the speech input. The splicing layer 220 may receive the speaker extraction embedding feature and the user's middle embedding voice feature, and output the splicing feature. The middle embedding voice feature of the user may be a pre-stored user feature, and may be for extracting information about a target speaker from a voice input. The second convolution layer 230 may receive a splicing feature and output a mask. The multiplier 240 may receive the mask and speaker extraction embedding features and output voice features of the target speaker.

According to an embodiment, the processor 110 may determine a speech scenario of the voice input based on original voice characteristics and voice characteristics of the target speaker. Speech scenarios of voice input may include a single speaker scenario and/or a multi-speaker scenario. The processor 110 may determine an utterance scenario of the voice input by comparing the original voice characteristics and the voice characteristics of the target speaker. The processor 110 may determine an utterance scenario as a single speaker scenario when a mean square error between the original voice feature and the target speaker's voice feature is smaller than a threshold value. The processor 110 may determine the speech scenario as a multi-speaker scenario when the mean square error between the original speech feature and the target speaker's speech feature is greater than or equal to a threshold value. Since the processor 110 may determine a speech scenario of the input voice based on the difference between the voice characteristics of the target speaker and the original voice characteristics, the speech scenario of the voice input may be easily determined.

According to an embodiment, the processor 110 may obtain final speech characteristics of the target speaker based on the determined speech scenario. In response to a single speaker scenario, the processor 110 may acquire final voice features of a target speaker by inputting original voice features to the second network 300 . In response to a multi-speaker scenario, the processor 110 may obtain final voice characteristics of the target speaker by inputting voice characteristics of the target speaker to the second network 300 . The final speech feature may have a fixed size regardless of the length of the speech. The final speech feature may be a one-dimensional vector. For example, the size of the final voice feature may be set to 1x128, 1x256, or 1x512.

According to an embodiment, the second network 300 may include a speaker embedding layer 310 and an attention statistics pooling layer 320 . The speaker embedding layer 310 may receive original voice features or voice features of the target speaker and output intermediate embedded voice features of the target speaker. The attention statistics pooling layer 320 may receive the middle embedded speech feature of the target speaker and output the final speech feature of the target speaker. The final speech feature may have a fixed size regardless of the length of the speech.

According to an embodiment, the processor 110 may determine whether the target speaker corresponds to the user based on the final voice characteristics of the target speaker and the final voice characteristics of the user. The processor 110 may calculate a similarity value between the final voice feature of the target speaker and the final voice feature of the user, and determine whether the target speaker corresponds to the user based on the calculated similarity value. For example, the processor 110 may determine that the target speaker corresponds to the user when a similarity value (eg, a cosine similarity value) is greater than a threshold value. For another example, when the similarity value is less than or equal to the threshold value, it may be determined that the target speaker does not correspond to the user. The processor 110 may provide a technique for accurately recognizing and verifying complex speech through an end-to-end model. The end-to-end model may include the first network 200 and the second network 300 described above. The first network 200 and the second network 300 may be jointly learned with a third network that converts the speaker's voice feature based on the speaker's intermediate embedded voice feature. Hereinafter, the learning operation of the networks will be described.

5 is an example for explaining learning of neural network-based models according to an embodiment, and FIG. 6 is another example for explaining learning of neural network-based models according to an embodiment.

Referring to FIG. 5 , the first network 510 , the second network 520 , and the third network 530 may be jointly learned. The first network 510 and the second network 520 may be trained by supervised learning, and the third network 530 may be trained by self-supervised learning. The first network 510 may be a model for extracting a speaker's voice. The second network 520 may be a model for performing speaker recognition. The third network may be a model for converting a speaker's voice. in the training process. The first network 510 may use a speaker separation loss function, the second network 520 may use a speaker recognition loss function, and the third network 530 may use a speech conversion loss function. The first network 510 , the second network 520 , and the third network 530 include speaker separation loss of the first network 510 , speaker recognition loss of the second network 520 , and third network 530 ) can be trained to minimize speech conversion loss. Since the end-to-end deep learning network is updated in consideration of speaker separation loss, speaker recognition loss, and speech conversion loss, the trained first network (eg, first network 200 in FIG. 2) and the trained second network (eg, : The second network 300 of FIG. 3 may have high recognition performance and generalization performance. It should be understood that the third network 530 is used in a training process to improve the performance of the first network 510 and the second network 520, but the third network 530 is not used in a recognition process.

A processor (eg, processor 110 of FIG. 1 ) may jointly train the first network 510 , the second network 520 , and the third network 530 using the original data set and the additional data set. there is. The original data set may include a plurality of voices and speaker tags respectively corresponding to the plurality of voices. Each of the plurality of voices may include a single speaker's voice.

Processor 110 may generate additional data sets based on the original data set. The additional data set may be a combination of at least two of a plurality of voices included in the original data set. For example, the processor 110 may select a first speaker's voice C and a second speaker's voice D from the original data set, and obtain voice M by combining the two voices. For example, voice M may be a direct fusion of voice C and voice D, and voice C may be used as a tag for voice M. An operation of tagging information corresponding to the newly generated voice may be automatically performed. That is, the additional data set may be automatically generated in the self-supervised learning process. Accordingly, the processor 110 can train the network without increasing the amount of data and annotations.

Optionally, the processor 110 may perform preprocessing (eg, random clipping, noise addition, reverberation addition, volume enhancement, etc.) on the generated additional data set.

The processor 110 may perform voice feature extraction through the feature extraction module 501 . For example, the processor 110 may extract original speech features of the original data set through the feature extraction module 501 and extract original speech features of the additional data set through the feature extraction module 501. .

The processor 110 may perform speaker recognition through the second network 520 . An input of the second network 520 may be an original voice feature extracted from a speaker's voice (eg, a voice included in the original data set). The speaker recognition loss function may be based on a metric learning loss. The processor 110 may update the second network 520 to minimize speaker recognition loss by adjusting the intra-class spacing.

The processor 110 may perform a speaker voice extraction task through the first network 510 . Inputs of the first network 510 may include original voice features extracted from multiple speakers' voices (eg, voices included in the additional data set) and intermediate embedded voice features of speakers output by the second network 520 . For example, the original speech feature may be extracted from the combined speech of the first speaker and the second speaker, and the intermediate embedded speech feature of the speaker may be a vector based on the first speaker's speech. In this case, the target speaker may be the first speaker. The processor 110 may update the second network 520 to minimize speaker separation loss by performing backpropagation.

The processor 110 may perform voice conversion through the third network 530 . The third network may include an Adaptive Instance Normalization (AdaIN) network for converting voice. An adaptive instance normalization network can receive two inputs. One of the inputs may be a voice feature to be converted, and the other of the inputs may be a voice feature to be converted. The adaptive instance normalization network can be expressed through Equation 1.

[Equation 1]

,

는 각각 변환의 대상인 음성 특징의 평균과 표준편차이고,

,

는 각각 변환의 기준이 되는 음성 특징의 평균과 표준편차일 수 있다. 예를 들어, 제3 네트워크(530)는 제1 화자의 음성에 기반한 벡터(예: 제1 화자의 중간 임베딩 음성 특징) 및 제1 화자의 음성 특징에 기초하여, 변환된 제1 화자의 음성 특징을 출력할 수 있다. 제1 화자의 음성에 기반한 벡터는 변환의 기준이 되는 음성 특징일 수 있고, 제1 화자의 음성 특징은 변환의 대상인 음성 특징일 수 있다. 프로세서(110)는 역전파를 수행함으로써, 음성 변환 손실을 최소화하도록 제3 네트워크(530)를 업데이트할 수 있다. 제3 네트워크(530)는 제2 네트워크(520)와 공동으로 업데이트될 수 있다.In Equation 1,

,

are the average and standard deviation of the speech features that are the target of transformation, respectively,

,

may be the average and standard deviation of speech features that are standards for conversion, respectively. For example, the third network 530 converts the voice feature of the first speaker based on the vector based on the voice of the first speaker (eg, the intermediate embedded voice feature of the first speaker) and the voice feature of the first speaker. can output The vector based on the voice of the first speaker may be a voice feature that is a standard for conversion, and the voice feature of the first speaker may be a voice feature that is a target for conversion. The processor 110 may update the third network 530 to minimize voice conversion loss by performing back propagation. The third network 530 may be jointly updated with the second network 520 .

Each network described with reference to FIG. 5 may be substantially similar to each network of FIG. 6 . That is, the description of each network described with reference to FIG. 5 may also be applied to each network of FIG. 6 . Therefore, redundant descriptions are omitted to avoid redundancy.

Referring to FIG. 6 , the network structure of FIG. 6 may be a parallel network structure. The network structure of FIG. 5 may be a serial network structure. Exemplary configurations of network structures (ie, series and parallel) are disclosed through FIGS. 5 and 6 , but the present invention is not limited thereto, and each network may be connected and/or configured in different ways.

7 is a flow diagram of an operation to identify a target speaker, according to one embodiment.

Operations 710 to 770 may be sequentially performed, but are not necessarily sequentially performed. For example, the order of each operation 710 to 770 may be changed, or at least two operations may be performed in parallel. Operations 710 to 770 may be performed by the aforementioned processor (eg, processor 110 of FIG. 1 ). Therefore, duplicate contents are omitted.

In operation 710, the processor 110 may extract voice features of the target speaker based on the voice input.

In operation 730, the processor 110 may determine an utterance scenario of the voice input based on the voice characteristics of the target speaker.

In operation 750, the processor 110 may obtain final speech characteristics of the target speaker based on the determined speech scenario.

In operation 770, the processor 110 may determine whether the target speaker corresponds to the user based on the final voice characteristics of the target speaker and the final voice characteristics of the user.

Although this disclosure includes specific examples, it is to those skilled in the art that various changes in form and detail can be made in these examples without departing from the spirit and scope of the claims and equivalents thereof. It will be clear. The examples described in the text are to be regarded for illustrative purposes only and not for limiting purposes. A description of a feature or aspect of each example is to be construed as applicable to similar features or aspects of other examples. Appropriate results may not be achieved if the described techniques are performed in a different order and/or if elements of the described system, architecture, device or circuit are otherwise combined and/or substituted or supplemented by other elements or their equivalents. can Therefore, the scope of the present invention is not limited by the specific examples, but is limited by the claims and their equivalents, and all changes within the scope of the claims and their equivalents should be construed as being included in the present invention. .

Claims (20)

In the operating method of the electronic device,
extracting speech characteristics of a target speaker based on a speech input;
determining a speech scenario of the voice input based on the voice characteristics of the target speaker;
obtaining a final speech feature of the target speaker based on the determined speech scenario; and
An operation of determining whether the target speaker corresponds to the user based on the final voice characteristics of the target speaker and the final voice characteristics of the user.
including,
The ignition scenario is
Including single speaker scenarios and multi-speaker scenarios,
Methods of operating electronic devices.
According to claim 1,
The extraction operation is
obtaining an original speech feature based on the speech input; and
An operation of extracting voice features of the target speaker by inputting the original voice features and the intermediate embedded voice features of the user to a first network.
Including, the operating method of the electronic device.
According to claim 2,
The first network,
a first convolution layer that receives the original speech features and outputs speaker extraction embedding features for extracting speech features of a speaker included in the speech input;
a splicing layer that receives the speaker extraction embedding feature and the user's middle embedding voice feature and outputs a splicing feature;
a second convolution layer that receives the splicing feature and outputs a mask; and
A multiplier receiving the mask and the speaker extraction embedding features and outputting speech features of the target speaker
Including, the operating method of the electronic device.
According to claim 3,
The operation of determining the speech scenario of the voice input,
An operation of determining a speech scenario of the voice input by comparing the original voice characteristics and the voice characteristics of the target speaker
Including, the operating method of the electronic device.
According to claim 4,
The operation of determining a speech scenario of the voice input by comparing the original voice characteristics and the voice characteristics of the target speaker,
determining the speech scenario as a single speaker scenario when a mean square error between the original speech feature and the target speaker's speech feature is less than a threshold value;
determining the speech scenario as a multi-speaker scenario when a mean square error between the original voice feature and the target speaker's voice feature is greater than or equal to a threshold value;
Including, the operating method of the electronic device.
According to claim 5,
The operation of obtaining the final speech feature of the target speaker,
corresponding to the single speaker scenario, acquiring final voice features of the target speaker by inputting the original voice features to a second network; or
In response to the multi-speaker scenario, obtaining a final voice feature of the target speaker by inputting the voice feature of the target speaker to the second network.
Including, the operating method of the electronic device.
According to claim 6,
The second network,
a speaker embedding layer which receives the original speech feature or the speech feature of the target speaker and outputs an intermediate embedded speech feature of the target speaker; and
An attention statistics pooling layer that receives the middle embedded speech feature of the target speaker and outputs the final speech feature of the target speaker.
Including, the operating method of the electronic device.
According to claim 7,
The operation of checking whether the target speaker corresponds to the user,
calculating a similarity value between the final voice feature of the target speaker and the final voice feature of the user; and
An operation of determining whether the target speaker corresponds to the user based on a calculation result
Including, the operating method of the electronic device.
According to claim 8,
The user's middle embedding speech characteristics are:
Acquired as a result of inputting the user's voice characteristics obtained based on the user's voice input to the speaker embedding layer,
The final voice characteristics of the user are:
Obtained as a result of inputting the user's intermediate embedding speech features to the attention statistics pooling layer,
Methods of operating electronic devices.
According to claim 9,
The first network and the second network
Learned jointly with a third network that transforms the speaker's speech features based on the speaker's intermediate embedded speech features;
Methods of operating electronic devices.
In electronic devices,
memory containing instructions; and
a processor electrically connected to the memory and configured to execute the instructions;
When the instructions are executed by the processor, the processor:
Extracting voice features of a target speaker based on the voice input;
determining an utterance scenario of the voice input based on the voice characteristics of the target speaker;
Obtain final speech characteristics of the target speaker based on the determined speech scenario;
determine whether the target speaker corresponds to the user based on the final voice characteristics of the target speaker and the final voice characteristics of the user;
The ignition scenario is
Including single speaker scenarios and multi-speaker scenarios,
electronic device.
According to claim 11,
the processor,
Obtain original speech features based on the speech input;
extracting voice features of the target speaker by inputting the original voice features and the intermediate embedded voice features of the user to a first network;
electronic device.
According to claim 12,
The first network,
a first convolution layer that receives the original speech features and outputs speaker extraction embedding features for extracting speech features of a speaker included in the speech input;
a splicing layer that receives the speaker extraction embedding feature and the user's middle embedding voice feature and outputs a splicing feature;
a second convolution layer that receives the splicing feature and outputs a mask; and
A multiplier receiving the mask and the speaker extraction embedding features and outputting speech features of the target speaker
Including, electronic device.
According to claim 13,
the processor,
determining a speech scenario of the voice input by comparing the original voice characteristics and the voice characteristics of the target speaker;
electronic device.
According to claim 14,
the processor,
determining the speech scenario as a single speaker scenario when a mean square error between the original speech feature and the target speaker's speech feature is less than a threshold value;
determining the speech scenario as a multi-speaker scenario when a mean square error between the original speech feature and the speech feature of the target speaker is greater than or equal to a threshold value;
electronic device.
According to claim 15,
the processor,
Corresponding to the single speaker scenario, the original speech feature is input to a second network to obtain the final speech feature of the target speaker; or
Corresponding to the multi-speaker scenario, inputting voice characteristics of the target speaker to the second network to obtain final voice characteristics of the target speaker.
electronic device.
According to claim 16,
The second network,
a speaker embedding layer which receives the original speech feature or the speech feature of the target speaker and outputs an intermediate embedded speech feature of the target speaker; and
An attention statistics pooling layer that receives the middle embedded speech feature of the target speaker and outputs the final speech feature of the target speaker.
Including, electronic device.
According to claim 17,
the processor,
calculating a similarity value between a final voice feature of the target speaker and a final voice feature of the user;
determining whether the target speaker corresponds to the user based on a calculation result;
electronic device.
According to claim 18,
The user's middle embedding speech characteristics are:
Acquired as a result of inputting the user's voice characteristics obtained based on the user's voice input to the speaker embedding layer,
The final voice characteristics of the user are:
Obtained as a result of inputting the user's intermediate embedding speech features to the attention statistics pooling layer,
electronic device.
According to claim 19,
The first network and the second network
Learned jointly with a third network that transforms the speaker's speech features based on the speaker's intermediate embedded speech features;
electronic device.

Images